Exploring and advising public sector reform across the ANZO region

TL;DR – I’ve taken on a new role, and this blog post explains why and how I came to the decision I did 🙂 Please scroll to the bottom or see my LinkedIn page if you just want to jump to the job.

If you’re reading this blog, you probably already know that I am passionate about and professionally committed to genuine and systemic reform/renewal of the public sector. I have spent the last decade in public service, and a decade before that in the tech sector, mostly focused on how to create better government services and policies, with digital/data transformation often providing a useful means for much needed reform.

I have written extensively about the many barriers facing systemic reform of the public sector, including on my blog. I know most public servants want change, but at this exact point in time there is a pressing need to:

  • create greater “demand” for genuine public sector renewal – demand is needed to prioritise investment in systemic reform, otherwise real transformation efforts become the can that continues to get kicked down the road, 
  • explore and invest in creating a more trustworthy public sector in a time of rapidly declining public trust, and 
  • explore what “good” could look like – especially in a world of rolling, continuous emergencies, where people understandably expect more of gov than ever before.

So when I left Service Canada, I realised I most want to focus on three key areas to address these pressing needs, to hopefully help make a real difference:

  1. A Transformed Public Sector: although we see a lot of effort around “service transformation”, there are still very few transformation programs that actually change the system. There are still significant barriers to create a genuinely adaptive, agile, human-centred, test driven and humane public sector, because it requires transformation of structures, budgetary approaches, program/project/product management, public engagement and culture across the sector, especially if we are to close the widening gap between policy and delivery. There is only rarely a planned or co-created future state being worked towards that would result in better, more equitable and more inclusive services and policies for all, and I suggest if you don’t have a clear and distinct future state, then you likely aren’t transforming but rather doing an iteration or digitisation of the current state. For example, RPA is usually used for just automating the status quo, and rarely a tool that enables process, policy, service or system redesign.
    Practically, I’d like to help shape future oriented, human-centred and adaptive policies, services, strategies and programs with governments.
  2. A Trustworthy Public Sector: public trust and confidence in public institutions is paramount to a stable and equitable society, but trust is in decline and communities are being gamed like never before. It is critical that public institutions take some time now to stop asking for trust (the “social licence” route), and to start focusing on how to be considered trustworthy. I think “trustworthy government” requires public participation as an important starting point but government systems and services also need to be reliable, traceable to their legal authority, testable against the law (regulation and legislation), audited in real time, easily understood by and appealable by citizens, and monitored for accountability. I’d basically like to see a modern implementation of Administrative Law coupled with a culture of openness, accountability and participatory governance with peer review, operational transparency and public participation. Finally, to be considered trustworthy, we need to collectively support the existential renewal of the sector, to support and demand a public service that is a long term steward for public good, not just a tool of the government of the day.
    Practically, I’ll be continuing to work on Legislation/Regulation as Code, explainable and high veracity systems, auditable and appealable systems, and of course, participatory governance/democracy initiatives. Imagine if we had a publicly available community repository of legislation as code to build upon and test against 🙂 What does a blend of participatory and representative democracy look like, that gets the best of both?
  3. An Augmented Public Sector: finally, I want to explore and support the timely concept of augmentation (rather than automation) in our public sector, whether it be service augmentation (“Alexa, I’ve lost my job, help me get support”), workforce augmentation (Hey Google, show me the human impact of this policy change across NSW”), or human augmentation (“I want to be the world’s best rock climber, so I might add a few more limbs for the Auglympics next year”). Augmentation to me is where a system purposefully supports humans and machines/tech to each do what they do best, without compromising our human values or the ethical and humane obligations we each have to each other. Too often I see “efficiency” projects that simply speed up or automate the status quo, or remove staff from a process, but we need to have better processes, more inclusive services, and more humane experiences with the public sector. There is no point speeding up the journey off a cliff 🙂 Rampant automation often undermines or misses the opportunity for better, more equitable, more ethical or more humane services and policies, so exploring and demonstrating the opportunities for service and workforce augmentation would be quite timely. Exploring and growing the understanding of human augmentation is also important as we will need to collectively deal with the potential implications for social cohesion when body hacking becomes more mainstream, given the human form is considered everything from sacred to irrelevant, depending on your culture and personal comfort or beliefs.
    Practically, this means exploring and demonstrating AI/AR/VR for the use cases above and getting away from pure automation uses of AI. Ideally in collaboration with others who are committed to more humane futures.

So, having decided these are the three areas I’d like to focus on, I had to consider where could I work to explore them? Where could possibly provide the breadth of opportunities to explore, build, influence, strategise and collaborate on public sector transformation? Sadly, many departments are pushed to stay within their wheelhouse and have become highly reactive to politics, and it is hard to drive a 5 or 10 year cross portfolio and systemic transformation agenda (let alone a 50 year vision!) within the constraints of an election cycle. Ideally, we need public institutions that are stable, operationally independent and confident stewards for long term public good, but this is a chicken and egg issue.

So I decided to work from the outside for a little while. A public service sabbatical of sorts, where I can contribute my expertise in the public domain, explore and demonstrate what “good” could look like, help build the demand and ambition for change (with executives, politics and the public), participate in community initiatives and grow my own experience. I plan to return to the sector to help drive systemic change when demand supports it 🙂

I’m pleased to say that after months of considering some very interesting and exciting options across different sectors, I was approached for a role as a Strategic Advisor with AWS, working with a small team called Strategic Development that works with the public sector across Australia, New Zealand and Oceania. Our focus is on supporting public institutions to dream big and explore new horizons, to achieve long term policy outcomes and sustainable public good, and to develop genuinely transformative plans with practical roadmaps to get there. As a team of accomplished public servants, we all understand the domain and want to help support and champion the sector and all public servants 🙂 I think I can both contribute a lot and learn a lot from this role, and it gives me a strong basis to drive my three objectives.

I will continue to work in the open, not just because it is my preference, but for peer review, collaboration and so you can all help me to keep it real and stay on track with the mission 🙂 Thanks to all those who advised me on this decision, and I hope to join a number of government advisory groups and boards where I can usefully contribute.

Please get in touch with me on piagov [at] amazon.com if you’d like to chat about any of the above! 🙂 I’m looking forward to collaborating on ambitious and transformative initiatives that create more mission oriented, values driven, humane and participatory public sectors across the region 🙂

In other news, my family had to return to Australia. Sick family + NZ travel policies at the time = having to move country again, but luckily the family member has recovered and we are now only a flight away from family 🙂 We decided if we were in Aus that we wanted to live somewhere glorious so we moved to Broome (WA), which has been just wonderful. I’ll be traveling for events and conferences, but am always available online and I’m looking forward to reconnecting with the Australian tech/data and public sectors after the last few years in NZ with the Canadian Government.

Sadly leaving the NSW Government

This week was sadly my last week with the NSW Government, Department of Customer Service, formerly the Department of Finance, Services and Innovation. I am sad to be leaving such an exciting place at such an exciting time, but after 12 months of commuting from Canberra to Sydney. The hardest part of working in the NSW Government has been, by far, the commute. I have been leaving my little family every week for 3, 4 or 5 days, and although we have explored possibilities to move, my family and I have to continue living in Canberra for the time being. It has got to the point where my almost 4 year old has asked me to choose her over work, a heart breaking scenario as many will understand. 

I wanted to publicly thank everyone I worked with, particularly my amazing teams who have put their heart, soul and minds to the task of making exceptional public services in an exceptional public sector. I am really proud of the two Branches I had the privilege and delight to lead, and I know whatever comes next, that those 160 or so individuals will continue to do great things wherever they go. 

I remain delighted and amazed at the unique opportunity in NSW Government to lead the way for truly innovative, holistic and user centred approaches to government. The commitment and leadership from William Murphy, Glenn King, Greg Wells, Damon Rees, Emma Hogan, Tim Reardon, Annette O’Callaghan, Michael Coutts-Trotter (and many others across the NSW Government senior executive) genuinely to my mind, has created the best conditions anywhere in Australia (and likely the world!) to make great and positive change in the public service.

I want to take a moment to also directly thank Martin Hoffman, Glenn, Greg, William, Amanda Ianna and all those who have supported me in the roles, as well as everyone from my two Branches over that 12 months for their support, belief and commitment. It has been a genuine privilege and delight to be a part of this exceptional department, and to see the incredible work across our Branches.

I have only been in the NSW Government for 12 months, and in that time was the ED for Digital Government Policy and Innovation for 9 months, and then ED Data, Insights and Transformation for a further 3 months.

In just 9 months, the Digital Government Policy and Innovation team achieved a lot in the NSW Government digital space, including:

  • Australia’s first Policy Lab (bringing agile test driven and user centred design methods into a traditional policy team),
  • the Digital Government Policy Landscape (mapping all digital gov policies for agencies) including IoT & a roadmap for an AI Ethics Framework and AI Strategy,
  • the NSW Government Digital Design Standard and a strong community of practice to contribute and collaborate
  • evolution of the Digital NSW Accelerator (DNA) to include delivery capabilities,
  • the School Online Enrolment system,
  • an operational and cross government Life Journeys Program (and subsequent life journey based navigators),
  • a world leading Rules as Code exemplars and early exploration of developing human and machine readable legislation from scratch(Better Rules),
  • establishment of a digital talent pool for NSW Gov,
  • great improvements to data.nsw and whole of government data policy and the Information Management Framework,
  • capability uplift across the NSW public sector including the Data Champions network and digital champions,
  • a prototype whole of government CX Pipeline,
  • the Innovation NSW team were recognised as one of Apolitical’s 100+ teams teaching government the skills of the future with a range of Innovation NSW projects including several Pitch to Pilot events, Future Economy breakfast series,
  • and the improvements to engagement/support we provided across whole of government.

For the last 3 months I was lucky to lead the newly formed and very exciting Data, Insights and Transformation Branch, which included the Data Analytics Centre, the Behavioural Insights Unit, and a new Transformation function to explore how we could design a modern public service fit for the 21st century. In only 3 months we

  • established a strong team culture, developed a clear cohesive work program, strategic objectives and service offerings,
  • chaired the ethics board for behavioural insights projects, which was a great experience, and
  • were seeing new interest, leads and engagement from agencies who wanted to engage with the Data Analytics Centre, Behavioural Insights Unit or our new Transformation function.

It was wonderful to work with such a fantastic group of people and I learned a lot, including from the incredible leadership team and my boss, William Murphy, who shared the following kind words about my leaving:

As a passionate advocate for digital and transformative approaches to deliver great public services, Pia has also been working steadily to deliver on whole-of-government approaches such as Government as a Platform, service analytics and our newly formed Transformation agenda to reimagine government.

Her unique and effective blend of systems thinking, technical creativity and vision will ensure the next stage in her career will be just as rewarding as her time with Customer Service has been.

Pia has made the difficult decision to leave Customer Service to spend more time with her Canberra-based family.

The great work Pia and her teams have done over the last twelve months has without a doubt set up the NSW digital and customer transformation agenda for success.

I want to thank her for the commitment and drive she has shown in her work with the NSW Government, and wish her well with her future endeavours. I’m confident her focus on building exceptional teams, her vision for NSW digital transformation and the relationships she has built across the sector will continue.

For my part, I’m not sure what will come next, but I’m going to have a holiday first to rest, and probably spend October simply writing down all my big ideas and doing some work on rules as code before I look for the next adventure.

Digital government: it all starts with open

This is a short video I did on the importance of openness for digital government, for the EngageTech Forum 2018. I’ve had a few people reuse it for other events so I thought I should blog it properly 🙂 Please see the transcript below. 

<Conference introductory remarks>

I wanted to talk about why openness and engagement is so critical for our work in a modern public service.

For me, looking at digital government, it’s not just about digital services, it’s about how we transform governments for the 21st century: how we do service delivery, engagement, collaboration, and how we do policy, legislation and regulation. How we make public services fit for purpose so they can serve you, the people, communities and economy of the 21st century.

For me, a lot of people think about digital and think about technology, but open government is a founding premise, a founding principle for digital government. Open that’s not digital doesn’t scale, and digital that’s not open doesn’t last. That doesn’t just mean looking at things like open source, open content and open APIs, but it means being open. Open to change. Being open to people and doing things with people, not just to people.

There’s a fundamental cultural, technical and process shift that we need to make, and it all starts with open.

<closing conference remarks>

Iteration or Transformation in government: paint jobs and engines

I was recently at an event talking about all things technology with a fascinating group of people. It was a reminder to me that digital transformation has become largely confused with digital iteration, and we need to reset the narrative around this space if we are to realise the real opportunities and benefits of technology moving forward. I gave a speech recently about major paradigm shifts that have brought us to where we are and I encourage everyone to consider and explore these paradigm shifts as important context for this blog post and their own work, but this blog post will focus specifically on a couple of examples of actual transformative change worth exploring.

The TL;DR is simply that you need to be careful to not mistake iteration for transformation. Iteration is an improvement on the status quo. Transformation is a new model of working that is, hopefully, fundamentally better than the status quo. As a rule of thumb, if what you are doing is simply better, faster or cheaper, that it is probably just iterative. There are many examples from innovation and digital transformation agendas which are just improvements on the status quo, but two examples of actual transformation of government I think are worth exploring are Gov-as-an-API and mutually beneficial partnerships to address shared challenges.

Background

Firstly, why am I even interested in “digital transformation”? Well, I’ve worked on open data in the Australian Federal Government since 2012 and very early on we recognised that open data was just a step towards the idea of “Gov as a Platform” as articulated by Tim O’Reilly nearly 10 years ago. Basically, he spoke about the potential to transform government into Government as a Platform, similar (for those unfamiliar with the “as a platform” idea) to Google Maps, or the Apple/Google app stores. Basically government could provide the data, content, transaction services and even business rules (regulation, common patterns such as means testing, building codes, etc) in a consumable, componentised and modular fashion to support a diverse ecosystem of service delivery, analysis and products by myriad agents, including private and public sector, but also citizens themselves.

Seems obvious right? I mean the private sector (the tech sector in any case) have been taking this approach for a decade.

What I have found in government is a lot of interest in “digital” where it is usually simply digitising an existing process, product or service. The understanding of consumable, modular architecture as a strategic approach to achieve greater flexibility and agility within an organisation, whilst enabling a broader ecosystem to build on top, is simply not understood by many. Certainly there are pockets that understand this, especially at the practitioner level, but agencies are naturally motivated to simply delivery what they need in isolation from a whole of government view. It was wonderful to recently see New Zealand picking up a whole of government approach in this vein but many governments are still focused on simple digitisation rather than transformation.

Why is this a problem? Well, to put it simply, government can’t scale the way it has traditionally worked to meet the needs and challenges of an increasingly changing world. Unless governments can transform to be more responsive, adaptive, collaborative and scalable, then they will become less relevant to the communities they serve and less effective in implementing government policy. Governments need to learn to adapt to the paradigm shifts from centrist to distributed models, from scarcity to surplus resources, from analogue to digital models, from command and control to collaborative relationships, and from closed to open practices.

Gov as an API

On of the greatest impacts of the DTO and the UK Government Digital Service has been to spur a race to the top around user centred design and agile across governments. However, these methods whilst necessary, are not sufficient for digital transformation, because you too easily see services created that are rapidly developed and better for citizens, but still based on bespoke siloed stacks of technology and content that aren’t reconsumable. Why does this matter? Because there are loads of components needed for multiple services, but siloed service technology stacks lead to duplication, a lack of agility in iterating and improving the user experience on an ongoing basis, a lack of programmatic access to those components which would enable system to system automation, and a complete lack of the “platform” upon which an ecosystem could be built.

When I was at the interim DTO in 2016, we fundamentally realised that no single agency would ever be naturally motivated, funded or mandated to deliver services on behalf of someone else. So rather than assuming a model wherein an agency is expected to do just that, we started considering new models. New systems wherein agencies could achieve what they needed (and were mandated and funded) to do, but where the broader ecosystem could provide multi-channel services delivery where there is no wrong door for citizens to do what they need. One channel might be the magical “life events” lens, another might be third parties, or State and Territory Governments, or citizen mashups. These agents and sectors have ongoing relationships with their users allowing them to exponentially spread and maintain user-centred design in way that government by itself can not afford to do, now or into the future.

This vision was itself was just a reflection of the Amazon, Google Maps, the Apple “apps store” and other platform models so prevalent in the private sector as described above. But governments everywhere have largely interpreted the “Gov as a Platform” idea as simply common or shared platforms. Whilst common platforms can provide savings and efficiencies, it has not enabled the system transformation needed to get true digital transformation across government.

So what does this mean practically? There are certainly pockets of people doing or experimenting in this space. Here are some of my thoughts to date based on work I’ve done in Australia (at the interim DTO) and in New Zealand (with the Department of Internal Affairs).

Firstly you can largely identify four categories of things involved in any government service:

  • Content – obvious, but taking into account the domain specific content of agencies as well as the kind of custodian or contextual content usually managed by points of aggregation or service delivery
  • Data – any type of list, source of intelligence or statistics, search queries such as ABN lookups
  • Transaction services – anything a person or business interacts with such as registration, payments, claims, reporting, etc. Obviously requires strict security frameworks
  • Business rules – the regulation, legislation, code, policy logic or even reusable patterns such as means testing which are usually hard coded into projects as required. Imagine an authoritative public API with the business logic of government available for consumption by everyone. A good example of pioneering work in this space is the Regulation as a Platform work by Data61.

These categories of components can all be made programmatically available for the delivery of your individual initiative and for broader reuse either publicly (for data, content and business rules) or securely (for transaction services). But you also need some core capabilities that are consumable for any form of digital service, below are a few to consider:

  • Identity and authentication, arguably also taking into account user consent based systems which may be provided from outside of government
  • Service analytics across digital and non digital channels to baseline the user experience and journey with govt and identify what works through evidence. This could also fuel a basic personalisation service.
  • A government web platform to pull together the government “sedan” service
  • Services register – a consumable register of government services (human services) to draw from across the board.

Imagine if we tool a conditional approach to matters, where you don’t need to provide documentation to prove your age (birth certificate, licence, passport), all of which give too much information, but rather can provide a verifiable claim that yes I am over the required age. This would both dramatically reduce the work for gov, and improve the privacy of people. See the verifiable claims work by W3C for more info on this concept, but it could be a huge transformation for how gov and privacy operates.

The three key advantages to taking this approach are:

  1. Agency agility – In splitting the front end from a consumable backend, agencies gain the ability to more rapidly iterate the customer experience of the service, taking into account changing user needs and new user platforms (mobile is just the start – augmented reality and embedded computing are just around the corner). When the back end and front end of a service are part of the one monolithic stack, it is simply too expensive and complicated to make many changes to the service.
  2. Ecosystem enablement – As identified above, a key game changer with the model is the ability for others to consume the services to support and multi-channel of services, analysis and products delivered by the broader community of government, industry and community players.
  3. Automation – the final and least sexy, though most interesting from a service improvement perspective, is automation. If your data, content, transaction systems and rules are programmatically available, suddenly you create the opportunity for the steps of a life event to be automated, where user consent is granted. The user consent part is really important, just to be clear! So rather than having 17 beautiful but distinct user services that a person has to individually complete, a user could be asked at any one of those entry points whether they’d like the other 16 steps to be automatically completed on their behalf. Perhaps the best way government can serve citizens in many cases is to get out of the way 🙂

Meaningful and mutually beneficial collaboration

Collaboration has become something of a buzzword in government often resulting in meetings, MOUs, principle statements or joint media releases. Occasionally there are genuine joint initiatives but there are still a lot of opportunities to explore new models of collaboration that achieve better outcomes.

Before we talk about how to collaborate, we need to address the elephant in the room: natural motivation. Government often sees consultation as something nice to have, collaboration as a nice way of getting others to contribute to something, and co-design as something to strive across the business units in your agency. If we consider the idea that government simply cannot meet the challenges or opportunities of the 21st century in isolation, if we acknowledge that government cannot scale at the same pace of the changing domains we serve, then we need to explore new models of collaboration where we actively partner with others for mutual benefit. To do this we need to identify areas for which others are naturally motivated to collaborate.

Firstly, let’s acknowledge there will always be work to do for which there are no naturally motivated partners. Why would anyone else want, at their own cost, to help you set up your mobility strategy, or implement an email server, or provide telephony services? The fact is that a reasonable amount of what any organisation does would be seen as BAU, as commodity, and thus only able to be delivered through internal capacity or contractual relationships with suppliers. So initiatives that try to improve government procurement practices can iteratively improve these customer-supplier arrangements but they don’t lend themselves to meaningful or significant collaboration.

OK, so what sort of things could be done differently? This is where you need to look critically at the purpose of your agency including the highest level goals, and identify who the natural potential allies in those goals could be. You can then approach your natural allies, identify where there are shared interests, challenges or opportunities, and collectively work together to co-design, co-invest, co-deliver and co-resource a better outcome for all involved. Individual allies could use their own resources or contractors for their contribution to the work, but the relationship is one of partnership, the effort and expertise is shared, and the outcomes are more powerful and effective than any one entity would have delivered on their own. In short, the whole becomes greater than the sum of its parts.

I will use the exciting and groundbreaking work of my current employer as a real example to demonstrate the point.

AUSTRAC is the Australian Government financial intelligence agency with some regulatory responsibilities. The purpose of the agency is threefold: 1) to detect and disrupt abuse of the financial system; 2) to strengthen the financial system against abuse; and 3) to contribute to the growth of the Australian economy. So who are natural allies in these goals… banks, law enforcement and fraud focused agencies, consumer protection organisations, regulatory organisations, fintech and regtech startups, international organisations, other governments, even individual citizens! So to tap into this ecosystem of potential allies, AUSTRAC has launched a new initiative called the “Fintel Alliance” which includes, at its heart, new models of collaborating on shared goals. There are joint intelligence operations on major investigations like the Panama Papers, joint industry initiatives to explore shared challenges and then develop prototypes and references implementations, active co-design of the new regulatory framework with industry, and international collaborations to strengthen the global financial system against abuse. The model is still in early days, but already AUSTRAC has shown that a small agency can punch well above it’s weight by working with others in new and innovative ways.

Other early DTO lessons

I’ll finish with a few lessons from the DTO. I worked at the DTO for the first 8 months (Jan – Sept 2015) when it was being set up. It was a crazy time with people from over 30 agencies thrust together to create a new vision for government services whilst simultaneously learning to speak each other’s language and think in a whole of government(s) way. We found a lot of interesting things, not least of all just how pervasive the siloed thinking of government ran. For example, internal analysis at the DTO of user research from across government agencies showed that user research tended to be through the narrow lens of an agency’s view of “it’s customers” and the services delivered by that agency. It was clear the user needs beyond the domain of the agency was seen as out of scope, or, at best, treated as a hand off point.

We started writing about a new draft vision whilst at the DTO which fundamentally was based on the idea of an evidence based, consumable approach to designing and delivering government services, built on reusable components that could be mashed up for a multi-channel ecosystem of service delivery. We tested this with users, agencies and industry with great feedback. Some of our early thinking is below, now a year and a half old, but worth referring back to:

One significant benefit of the DTO and GDS was the cycling of public servants through the agency to experience new ways of working and thinking, and applying an all of government lens across their work. This cultural transformation was then maintained in Australia, at least in part, when those individuals returned to their home agencies. A great lesson for others in this space.

A couple of other lessons learned from the DTO are below:

  • Agencies want to change. They are under pressure from citizens, governments and under budget constraints and know they need better ways to do things.
  • A sandbox is important. Agencies need somewhere to experiment, play with new tools, ideas and methods, draw on different expertise and perspectives, build prototypes and try new ideas. This is ideally best used before major projects are undertaken as a way to quickly test ideas before going to market. It also helps improve expectations of what is possible and what things should cost.
  • Everyone has an agenda, every agency will drive their own agenda with whatever the language of the day and agendas will continue to diverge from each other whilst there is not common vision.
  • Evidence is important! And there isn’t generally enough AoG evidence available. Creating an evidence base was a critical part of identifying what works and what doesn’t.
  • Agile is a very specific and useful methodology, but often gets interpreted as something loose, fast, and unreliable. Education about proper agile methods is important.
  • An AoG strategy for transformation is critical. If transformation is seen as a side project, it will never be integrated into BAU.
  • Internal brilliance needs tapping. Too often govt brings in consultants and ignores internal ideas, skills and enthusiasm. There needs to be a combination of public engagement and internal engagement to get the best outcomes.

I want to just finish by acknowledging and thanking the “interim DTO” team and early leadership for their amazing work, vision and collective efforts in establishing the DTO and imagining a better future for service delivery and for government more broadly. It was an incredible time with incredible people, and your work continues to live on and be validated by service delivery initiatives in Australia and across the world. Particular kudos to team I worked directly with, innovative and awesome public servants all! Sharyn Clarkson, Sean Minney, Mark Muir, Vanessa Roarty, Monique Kenningham, Nigel O’Keefe, Mark McKenzie, Chris Gough, Deb Blackburn, Lisa Howdin, Simon Fisher, Andrew Carter, Fran Ballard and Fiona Payne 🙂 Also to our contractors at the time Ruth Ellison, Donna Spencer and of course, the incredible and awesome Alex Sadleir.

Personal submission to the Productivity Commission Review on Public Sector Data

My name is Pia Waugh and this is my personal submission to the Productivity Commission Review on Public Sector Data. It does not reflect the priorities or agenda of my employers past, present or future, though it does draw on my expertise and experience in driving the open data agenda and running data portals in the ACT and Commonwealth Governments from 2011 till 2015. I was invited by the Productivity Commission to do a submission and thought I could provide some useful ideas for consideration. I note I have been on maternity leave since January 2016 and am not employed by the Department of Prime Minister and Cabinet or working on data.gov.au at the time of writing this submission. This submission is also influenced by my work and collaboration with other Government jurisdictions across Australia, overseas and various organisations in the private and community sectors. I’m more than happy to discuss these ideas or others if useful to the Productivity Commission.

I would like to thank all those program and policy managers, civic hackers, experts, advocates, data publishers, data users, public servants and vendors whom I have had the pleasure to work with and have contributed to my understanding of this space. I’d also like to say a very special thank you to the Australian Government Chief Technology Officer, John Sheridan, who gave me the freedom to do what was needed with data.gov.au, and to Allan Barger who was my right hand man in rebooting the agenda in 2013, supporting agencies and helping establish a culture of data publishing and sharing across the public sector. I think we achieved a lot in only a few years with a very small but highly skilled team. A big thank you also to Alex Sadleir and Steven De Costa who were great to work with and made it easy to have an agile and responsive approach to building the foundation for an important piece of data infrastructure for the Australian Government.

Finally, this is a collection of some of my ideas and feedback for use by the Productivity Commission however, it doesn’t include everything I could possibly have to say on this topic because, frankly, we have a small baby who is taking most of my time at the moment. Please feel free to add your comments, criticisms or other ideas to the comments below! It is all licensed as Creative Commons 4.0 By Attribution, so I hope it is useful to others working in this space.

The Importance of Vision

Without a vision, we stumble blindly in the darkness. Without a vision, the work and behaviours of people and organisations are inevitably driven by other competing and often short term priorities. In the case of large and complex organisms like the Australian Public Service, if there is no cohesive vision, no clear goal to aim for, then each individual department is going to do things their own way, driven by their own priorities, budgets, Ministerial whims and you end up with what we largely have today: a cacophony of increasingly divergent approaches driven by tribalism that make collaboration, interoperability, common systems and data reuse impossible (or prohibitively expensive).

If however, you can establish a common vision, then even a strongly decentralised system can converge on the goal. If we can establish a common vision for public data, then the implementation of data programs and policies across the APS should become naturally more consistent and common in practice, with people naturally motivated to collaborate, to share expertise, and to reuse systems, standards and approaches in pursuit of the same goal.

My vision for public data is two-pronged and a bit of a paradigm shift: data by design and gov as an API! “Data by design” is about taking a data driven approach to the business of government and “gov as an API” is about changing the way we use, consume, publish and share data to properly enable a data driven public service and a broader network of innovation. The implementation of these ideas would create mashable government that could span departments, jurisdictions and international boundaries. In a heavily globalised world, no government is in isolation and it is only by making government data, content and services API enabled and reusable/interfacable, that we, collectively, can start to build the kind of analysis, products and services that meet the necessarily cross jurisdictional needs of all Australians, of all people.

More specifically, my vision is a data driven approach to the entire business of government that supports:

  • evidence based and iterative policy making and implementation;

  • transparent, accountable and responsible Government;

  • an open competitive marketplace built on mashable government data, content and services; and

  • a more efficient, effective and responsive public service.

What this requires is not so simple, but is utterly achievable if we could embed a more holistic whole of government approach in the work of individual departments, and then identify and fill the gaps through a central capacity that is responsible for driving a whole of government approach. Too often we see the data agenda oversimplified into what outcomes are desired (data visualisations, dashboards, analysis, etc) however, it is only in establishing multipurpose data infrastructure which can be reused for many different purposes that we will enable the kind of insights, innovation, efficiencies and effectiveness that all the latest reports on realising the value of data allude to. Without actual data, all the reports, policies, mission statements, programs and governance committees are essentially wasting time. But to get better government data, we need to build capacity and motivation in the public sector. We need to build a data driven culture in government. We also need to grow consumer confidence because a) demand helps drive supply, and b) if data users outside the public sector don’t trust that they can find, use and rely upon at least some government data, then we won’t ever see serious reuse of government data by the private sector, researchers, non-profits, citizens or the broader community.

Below is a quick breakdown of each of these priorities, followed by specific recommendations for each:

data infrastructure that supports multiple types of reuse. Ideally all data infrastructure developed by all government entities should be built in a modular, API enabled way to support data reuse beyond the original purpose to enable greater sharing, analysis, aggregation (where required) and publishing. It is often hard for agencies to know what common infrastructure already exists and it is easy for gaps to emerge, so another part of this is to map the data infrastructure requirements for all government data purposes, identify where solutions exist and any gaps. Where whole of government approaches are identified, common data infrastructure should be made available for whole of government use, to reduce the barrier to publishing and sharing data for departments. Too often, large expensive data projects are implemented in individual agencies as single purpose analytics solutions that don’t make the underlying data accessible for any other purpose. If such projects separated the data infrastructure from the analytics solutions, then the data infrastructure could be built to support myriad reuse including multiple analytics solutions, aggregation, sharing and publishing. If government data infrastructure was built like any other national infrastructure, it should enable a competitive marketplace of analysis, products and service delivery both domestically and globally. A useful analogy to consider is the example of roads. Roads are not typically built just from one address to another and are certainly not made to only support certain types of vehicles. It would be extremely inefficient if everyone built their own custom roads and then had to build custom vehicles for each type of road. It is more efficient to build common roads to a minimum technical standard that any type of vehicle can use to support both immediate transport needs, but also unknown transport needs into the future. Similarly we need to build multipurpose data infrastructure to support many types of uses.

greater publisher capacity and motivation to share and publish data. Currently the range of publishing capacity across the APS is extremely broad, from agencies that do nothing to agencies that are prolific publishers. This is driven primarily by different cultures and responsibilities of agencies and if we are to improve the use of data, we need to improve the supply of data across the entire public sector. This means education and support for agencies to help them understand the value to their BAU work. The time and money saved by publishing data, opportunities to improve data quality, the innovation opportunities and the ability to improve decision making are all great motivations once understood, but generally the data agenda is only pitched in political terms that have little to no meaning to data publishers. Otherwise there is no natural motivation to publish or share data, and the strongest policy or regulation in the world does not create sustainable change or effective outcomes if you cannot establish a motivation to comply. Whilst ever publishing data is seen as merely a compliance issue, it will be unlikely for agencies to invest the time and skills to publish data well, that is, to publish the sort of data that consumers want to use.

greater consumer confidence to improve the value realised from government data. Supply is nothing without demand and currently there is a relatively small (but growing) demand for government data, largely because people won’t use what they don’t trust. In the current landscape is difficult to find data and even if one can find it, it is often not machine readable or not freely available, is out of date and generally hard to use. There is not a high level of consumer confidence in what is provided by government so many people don’t even bother to look. If they do look, they find myriad data sources of ranging quality and inevitably waste many hours trying to get an outcome. There is a reasonable demand for data for research and the research community tends to jump through hoops – albeit reluctantly and at great cost – to gain access to government data. However, the private and civic sectors are yet to seriously engage apart form a few interesting outliers. We need to make finding and using useful data easy, and start to build consumer confidence or we will never even scratch the surface of the billions of dollars of untapped potential predicted by various studies. The data infrastructure section is obviously an important part of building consumer confidence as it should make it easier for consumers to find and have confidence in what they need, but it also requires improving the data culture across the APS, better outreach and communications, better education for public servants and citizens on how to engage in the agenda, and targeted programs to improve the publishing of data already in demand. What we don’t need is yet another “tell us what data you want” because people want to see progress.

a data driven culture that embeds in all public servants an understanding of the role of data in the every day work of the public service, from program management, policy development, regulation and even basic reporting. It is important to take data from being seen as a specialist niche delegated only to highly specialised teams and put data front and centre as part of the responsibilities of all public servants – especially management – in their BAU activities. Developing this culture requires education, data driven requirements for new programs and policies, some basic skills development but mostly the proliferation of an awareness of what data is, why it is important, and how to engage appropriate data skills in the BAU work to ensure a data driven approach. Only with data can a truly evidence driven approach to policy be taken, and only with data can a meaningful iterative approach be taken over time.

Finally, obviously the approach above requires an appropriately skilled team to drive policy, coordination and implementation of the agenda in collaboration with the broader APS. This team should reside in a central agenda to have whole of government imprimatur, and needs a mix of policy, commercial, engagement and technical data skills. The experience of data programs around the world shows that when you split policy and implementation, you inevitably get both a policy team lacking in the expertise to drive meaningful policy and an implementation team paralysed by policy indecision and an unclear mandate. This space is changing so rapidly that policy and implementation need to be agile and mutually reinforcing with a strong focus on getting things done.

As we examine the interesting opportunities presented by new developments such as blockchain and big data, we need to seriously understand the shift in paradigm from scarcity to surplus, from centralised to distributed systems, and from pre-planned to iterative approaches, if we are to create an effective public service for the 21st century.

There is already a lot of good work happening, so the recommendations in this submission are meant to improve and augment the landscape, not replicate. I will leave areas of specialisation to the specialists, and have tried to make recommendations that are supportive of a holistic approach to developing a data-driven public service in Australia.

Current Landscape

There has been progress in recent years towards a more data driven public sector however, these initiatives tend to be done by individual teams in isolation from the broader public service. Although we have seen some excellent exemplars of big data and open data, and some good work to clarify and communicate the intent of a data driven public service through policy and reviews, most projects have simply expanded upon the status quo thinking of government as a series of heavily fortified castles that take the extraordinary effort of letting in outsiders (including other departments) only under strictly controlled conditions and with great reluctance and cost. There is very little sharing at the implementation level (though an increasing amount of sharing of ideas and experience) and very rarely are new initiatives consulted across the APS for a whole of government perspective. Very rarely are actual data and infrastructure experts encouraged or supported to work directly together across agency or jurisdiction lines, which is a great pity. Although we have seen the idea of the value of data start to be realised and prioritised, we still see the implementation of data projects largely delegated to small, overworked and highly specialised internal teams that are largely not in the habit of collaborating externally and thus there is a lot of reinvention and diversity in what is done.

If we are to realise the real benefits of data in government and the broader economy, we need to challenge some of the status quo thinking and approaches towards data. We need to consider government (and the data it collects) as a platform for others to build upon rather than the delivery mechanism for all things to all people. We also need to better map what is needed for a data-driven public service rather than falling victim to the attractive (and common, and cheap) notion of simply identifying existing programs of work and claiming them to be sufficient to meet the goals of the agenda.

Globally this is still a fairly new space. Certain data specialisations have matured in government (eg. census/statistics, some spatial, some science data) but there is still a lack of a cohesive approach to data in any one agency. Even specialist data agencies tend to not look beyond the specialised data to have a holistic data driven approach to everything. In this way, it is critical to develop a holistic approach to data at all levels of the public service to embed the principles of data driven decision making in everything we do. Catalogues are not enough. Specialist data projects are not enough. Publishing data isn’t enough. Reporting number of datasets quickly becomes meaningless. We need to measure our success in this space by how well data is helping the public service to make better decisions, build better services, develop and iterate responsive and evidence based policy agendas, measure progress and understand the environment in which we operate.

Ideally, government agencies need to adopt a dramatic shift in thinking to assume in the first instance that the best results will be discovered through collaboration, through sharing, through helping people help themselves. There also needs in the APS to be a shift away from thinking that a policy, framework, governance structure or other artificial constructs are sufficient outcomes. Such mechanisms can be useful, but they can also be a distraction from getting anything tangible done. Such mechanisms often add layers of complexity and cost to what they purport to achieve. Ultimately, it is only what is actually implemented that will drive an outcome and I strongly believe an outcomes driven approach must be applied to the public data agenda for it to achieve its potential.

References

In recent years there has been a lot of progress. Below is a quick list to ensure they are known and built upon for the future. It is also useful to recognise the good work of the government agencies to date.

  • Public Data Toolkit – the data.gov.au team have pulled together a large repository of information, guidance and reports over the past 3 years on our open data toolkit at http://toolkit.data.gov.au. There are also some useful contributions from the Department of Communications Spatial Policy Branch. The Toolkit has links to various guidance from different authoritative agencies across the APS as well as general information about data management and publishing which would be useful to this review.

  • The Productivity Commission is already aware of the Legislative and other Barriers Workshop I ran at PM&C before going on maternity leave, and I commend the outcomes of that session to the Review.

  • The Financial Sector Inquiry (the “Murray Inquiry”) has some excellent recommendations regarding the use of data-drive approaches to streamline the work and reporting of the public sector which, if implemented, would generate cost and time savings as well as the useful side effect of putting in place data driven practices and approaches which can be further leveraged for other purposes.

  • Gov 2.0 Report and the Ahead of the Game Report – these are hard to find copies of online now, but have some good recommendations and ideas about a more data centric and evidence based public sector and I commend them both to the Review. I’m happy to provide copies if required.

  • There are many notable APS agency efforts which I recommend the Productivity Commission engage with, if they haven’t already. Below are a few I have come across to date, and it is far from an exhaustive list:

    • PM&C (Public Data Management Report/Implementation & Public Data Policy Statement)

    • Finance (running and rebooting data.gov.au, budget publishing, data integration in GovCMS)

    • ABS (multi agency arrangement, ABS.Stat)

    • DHS (analytics skills program, data infrastructure and analysis work)

    • Immigration (analytics and data publishing)

    • Social Services (benefits of data publishing)

    • Treasury (Budget work)

    • ANDS (catalogue work and upskilling in research sector)

    • NDI (super computer functionality for science)

    • ATO (smarter data program, automated and publications data publishing, service analytics, analytics, dev lab, innovationspace)

    • Industry (Lighthouse data integration and analysis, energy ratings data and app)

    • CrimTRAC and AUSTRAC (data collection, consolidation, analysis, sharing)

  • Other jurisdictions in Australia have done excellent work as well and you can see a list (hopefully up to date) of portals and policies on the Public Data Toolkit. I recommend the Productivity Commission engage with the various data teams for their experiences and expertise in this matter. There are outstanding efforts in all the State and Territory Governments involved as well as many Local Councils with instructive success stories, excellent approaches to policy, implementation and agency engagement/skills and private sector engagement projects.

Som current risks/issues

There are a number of issues and risks that exist in pursuing the current approach to data in the APS. Below are some considerations to take into account with any new policies or agendas to be developed.

  • There is significant duplication of infrastructure and investment from building bespoke analytics solutions rather than reusable data infrastructure that could support multiple analytics solutions. Agencies build multiple bespoke analytics projects without making the underpinning data available for other purposes resulting in duplicated efforts and under-utilised data across government.

  • Too much focus on pretty user interfaces without enough significant investment or focus on data delivery.

  • Discovery versus reuse – too many example of catalogues linking to dead data. Without the data, discovery is less than useful.

  • Limitations of tech in agencies by ICT Department – often the ICT Department in an agency is reticent to expand the standard operating environment beyond the status quo, creating an issue of limitation of tools and new technologies.

  • Copyright and legislation – particularly old interpretations of each and other excuses to not share.

  • Blockers to agencies publishing data (skills, resources, time, legislation, tech, competing priorities e.g. assumed to be only specialists that can do data).

  • Often activities in the public sector are designed to maintain the status quo (budgets, responsibilities, staff count) and there is very little motivation to do things more efficiently or effectively. We need to establish these motivations for any chance to be sustainable.

  • Public perceptions about the roles and responsibilities of government change over time and it is important to stay engaged when governments want to try something new that the public might be uncertain about. There has been a lot of media attention about how data is used by government with concerns aired about privacy. Australians are concerned about what Government plans to do with their data. Broadly the Government needs to understand and engage with the public about what data it holds and how it is used. There needs to be trust built to both improve the benefits from data and to ensure citizen privacy and rights are protected. Where government wants to use data in new ways, it needs to prosecute the case with the public and ensure there are appropriate limitations to use in place to avoid misuse of the data. Generally, where Australians can directly view the benefit of their data being used and where appropriate limitations are in place, they will probably react positively. For example, tax submission are easier now that their data auto-fills from their employers and health providers when completing Online Tax. People appreciate the concept of having to only update their details once with government.

Benefits

I agree with the benefits identified by the Productivity Commission discussion paper however I would add the following:

  • Publishing government data, if published well, enables a competitive marketplace of service and product delivery, the ability to better leverage public and academic analysis for government use and more broadly, taps into the natural motivation of the entire community to innovate, solve problems and improve life.

  • Establishing authoritative data – often government is the authoritative source of information it naturally collects as part of the function of government. When this data is not then publicly available (through anonymised APIs if necessary) then people will use whatever data they can get access to, reducing the authority of the data collected by Government

  • A data-drive approach to collecting, sharing and publishing data enables true iterative approaches to policy and services. Without data, any changes to policy are difficult to justify and impossible to track the impact, so data provides a means to support change and to identify what is working quickly. Such feedback loops enable iterative improvements to policies and programs that can respond to the changing financial and social environment the operate in.

  • Publishing information in a data driven way can dramatically streamline reporting, government processes and decision making, freeing up resources that can be used for more high value purposes.

Public Sector Data Principles

The Public Data Statement provides a good basis of principles for this agenda. Below are some principles I think are useful to highlight with a brief explanation of each.

Principles:

  • build for the future – legacy systems will always be harder to deal with so agencies need to draw a line in the sand and ensure new systems are designed with data principles, future reuse and this policy agenda in mind. Otherwise we will continue to build legacy systems into the future. Meanwhile, just because a legacy system doesn’t natively support APIs or improved access doesn’t mean you can’t affordably build middleware solutions to extract, transform, share and publish data in an automated way.

  • data first – wherever data is used to achieve an outcome, publish the data along with the outcome. This will improve public confidence in government outcomes and will also enable greater reuse of government data. For example, where graphs or analysis are published also publish the data. Where a mobile app is using data, publish the data API. Where a dashboard is set up, also provide access to the underpinning data.

  • use existing data, from the source where possible – this may involve engaging with or even paying for data from private sector or NGOs, negotiating with other jurisdictions or simply working with other government entities to gain access.

  • build reusable data infrastructure first – wherever data is part of a solution, the data should be accessible through APIs so that other outcomes and uses can be realised, even if the APIs are only used for internal access in the first instance.

  • data driven decision making to support iterative and responsive policy and implementations approaches – all decisions should be evidence based, all projects, policies and programs should have useful data indicators identified to measure and monitor the initiative and enable iterative changes backed by evidence.

  • consume your own data and APIs – agencies should consider how they can better use their own data assets and build access models for their own use that can be used publicly where possible. In consuming their own data and APIs, there is a better chance the data and APIs will be designed and maintained to support reliable reuse. This could raw or aggregate data APIs for analytics, dashboards, mobile apps, websites, publications, data visualisations or any other purpose.

  • developer empathy – if government agencies start to prioritise the needs of data users when publishing data, there is a far greater likelihood the data will be published in a way developers can use. For instance, no developer likes to use PDFs, so why would an agency publish data in a PDF (hint: there is no valid reason. PDF does not make your data more secure!).

  • standardise where beneficial but don’t allow the perfect to delay the good – often the focus on data jumps straight to standards and then multi year/decade standards initiatives are stood up which creates huge delays to accessing actual data. If data is machine readable, it can often be used and mapped to some degree which is useful, more useful than having access to nothing.

  • automate, automate, automate! – where human effort is required, tasks will always be inefficient and prone to error. Data collection, sharing and publishing should be automated where possible. For example, when data is regularly requested, agencies should automate the publishing of data and updates which both reduces the work for the agency and improves the quality for data users.

  • common platforms – where possible agencies should use existing common platforms to share and publish data. Where they need to develop new infrastructure, efforts should be made to identify where new platforms might be useful in a whole of government or multi agency context and built to be shared. This will support greater reuse of infrastructure as well as data.

  • a little less conversation a little more action – the public service needs to shift from talking about data to doing more in this space. Pilot projects, experimentation, collaboration between implementation teams and practitioners, and generally a greater focus on getting things done.

Recommendations for the Public Data agenda

Strategic

  1. Strong Recommendation: Develop a holistic vision and strategy for a data-driven APS. This could perhaps be part of a broader digital or ICT strategy, but there needs to be a clear goal that all government entities are aiming towards. Otherwise each agency will continue to do whatever they think makes sense just for them with no convergence in approach and no motivation to work together.

  2. Strong Recommendation: Develop and publish work program and roadmap with meaningful measures of progress and success regularly reported publicly on a public data agenda dashboard. NSW Government already have a public roadmap and dashboard to report progress on their open data agenda.

Whole of government data infrastructure

  1. Strong Recommendation: Grow the data.gov.au technical team to at least 5 people to grow the whole of government catalogue and cloud based data hosting infrastructure, to grow functionality in response to data publisher and data user requirements, to provide free technical support and training to agencies, and to regularly engage with data users to grow public confidence in government data. The data.gov.au experience demonstrated that even just a small motivated technical team could greatly assist agencies to start on their data publishing journey to move beyond policy hypothesising into practical implementation. This is not something that can be efficiently or effectively outsourced in my experience.

  • I note that in the latest report from PM&C, Data61 have been engaged to improve the infrastructure (which looks quite interesting) however, there still needs to be an internal technical capability to work collaboratively with Data61, to support agencies, to ensure what is delivered by contractors meets the technical needs of government, to understand and continually improve the technical needs and landscape of the APS, to contribute meaningfully to programs and initiatives by other agencies, and to ensure the policies and programs of the Public Data Branch are informed by technical realities.

  1. Recommendation: Establish/extend a data infrastructure governance/oversight group with representatives from all major data infrastructure provider agencies including the central public data team to improve alignment of agendas and approaches for a more holistic whole of government approach to all major data infrastructure projects. The group would assess new data functional requirements identified over time, would identify how to best collectively meet the changing data needs of the public sector and would ensure that major data projects apply appropriate principles and policies to enable a data driven public service. This work would also need to be aligned with the work of the Data Champions Network.

  2. Recommendation: Map out, publish and keep up to date the data infrastructure landscape to assist agencies in finding and using common platforms.

  3. Recommendation: Identify on an ongoing basis publisher needs and provide whole of government solutions where required to support data sharing and publishing (eg – data.gov.au, ABS infrastructure, NationalMap, analytics tools, github and code for automation, whole of gov arrangements for common tools where they provide cost benefits).

  4. Recommendation: Create a requirement for New Policy Proposals that any major data initiatives (particularly analytics projects) also make the data available via accessible APIs to support other uses or publishing of the data.

  5. Recommendation: Establish (or build upon existing efforts) an experimental data playground or series of playgrounds for agencies to freely experiment with data, develop skills, trial new tools and approaches to data management, sharing, publishing, analysis and reuse. There are already some sandbox environments available and these could be mapped and updated over time for agencies to easily find and engage with such initiatives.

Grow consumer confidence

  1. Strong Recommendation: Build automated data quality indicators into data.gov.au. Public quality indicators provide an easy way to identify quality data, thus reducing the time and effort required by data users to find something useful. This could also support a quality search interface, for instance data users could limit searches to “high quality government data” or choose granular options such as “select data updated this year”. See my earlier blog (from PM&C) draft of basic technical quality indicators which would be implemented quickly, giving data users a basic indication of how usable and useful data is in a consistent automated way. Additional quality indicators including domain specific quality indicators could be implemented in a second or subsequent iteration of the framework.

  2. Strong Recommendation: Establish regular public communications and engagement to improve relations with data users, improve perception of agenda and progress and identify areas of data provision to prioritise. Monthly blogging of progress, public access to the agenda roadmap and reporting on progress would all be useful. Silence is generally assumed to mean stagnation, so it is imperative for this agenda to have a strong public profile, which in part relies upon people increasingly using government data.

  3. Strong Recommendation: Establish a reasonable funding pool for agencies to apply for when establishing new data infrastructure, when trying to make existing legacy systems more data friendly, and for responding to public data requests in a timely fashion. Agencies should also be able to apply for specialist resource sharing from the central and other agencies for such projects. This will create the capacity to respond to public needs faster and develop skills across the APS.

  4. Strong Recommendation: The Australian Government undertake an intensive study to understand the concerns Australians hold relating to the use of their data and develop a new social pact with the public regarding the use and limitations of data.

  5. Recommendation: establish a 1-2 year project to support Finance in implementing the data driven recommendations from the Murray Inquiry with 2-3 dedicated technical resources working with relevant agency teams. This will result in regulatory streamlining, improved reporting and analysis across the APS, reduced cost and effort in the regular reporting requirements of government entities and greater reuse of the data generated by government reporting.

  6. Recommendation: Establish short program to focus on publishing and reporting progress on some useful high value datasets, applying the Public Data Policy Statement requirements for data publishing. The list of high value datasets could be drawn from the Data Barometer, the Murray Inquiry, existing requests from data.gov.au, and work from PM&C. The effort of determining the MOST high value data to publish has potentially got in the way of actual publishing, so it would be better to use existing analysis and prioritise some data sets but more importantly to establish data by default approach across govt for the kinds of serendipitous use of data for truly innovation outcomes.

  7. Recommendation: Citizen driven privacy – give citizens the option to share data for benefits and simplified services, and a way to access data about themselves.

Grow publisher capacity and motivation

  1. Strong Recommendation: Document the benefits for agencies to share data and create better guidance for agencies. There has been a lot of work since the reboot of data.gov.au to educate agencies on the value of publishing data. The value of specialised data sharing and analytics projects is often evident driving those kinds of projects, but traditionally there hasn’t been a lot of natural motivations for agencies to publish data, which had the unfortunate result of low levels of data publishing. There is a lot of anecdotal evidence that agencies have saved time and money by publishing data publicly, which have in turn driven greater engagement and improvements in data publishing by agencies. If these examples were better documented (now that there are more resources) and if agencies were given more support in developing holistic public data strategies, we would likely see more data published by agencies.

  2. Strong Recommendation: Implement an Agency League Table to show agency performance on publishing or otherwise making government data publicly available. I believe such a league table needs to be carefully designed to include measures that will drive better behaviours in this space. I have previously mapped out a draft league table which ranks agency performance by quantity (number of data resources, weighted by type), quality (see previous note on quality metrics), efficiency (the time and/or money saved in publishing data) and value (a weighted measure of usage and reuse case studies) and would be happy to work with others in re-designing the best approach if useful.

  3. Recommendation: Establish regular internal hackfests with tools for agencies to experiment with new approaches to data collection, sharing, publishing and analysis – build on ATO lab, cloud tools, ATO research week, etc.

  4. Recommendation: Require data reporting component for New Policy Proposals and new tech projects wherein meaningful data and metrics are identified that will provide intelligence on the progress of the initiative throughout the entire process, not just at the end of the project.

  5. Recommendation: Add data principles and API driven and automated data provision to the digital service standard and APSC training.

  6. Recommendation: Require public APIs for all government data, appropriately aggregated where required, leveraging common infrastructure where possible.

  7. Recommendation: Establish a “policy difference engine” – a policy dashboard that tracks the top 10 or 20 policy objectives for the government of the day which includes meaningful metrics for each policy objective over time. This will enable the discovery of trends, the identification of whether policies are meeting their objectives, and supports an evidence based iterative approach to the policies because the difference made by any tweaks to the policy agenda will be evident.

  8. Recommendation: all publicly funded research data to be published publicly, and discoverable on central research data hub with free hosting available for research institutions. There has been a lot of work by ANDS and various research institutions to improve discovery of research data, but a large proportion is still only available behind a paywall or with an education logon. A central repository would reduce the barrier for research organisations to publicly publish their data.

  9. Recommendation: Require that major ICT and data initiatives consider cloud environments for the provision, hosting or analysis of data.

  10. Recommendation: Identify and then extend or provide commonly required spatial web services to support agencies in spatially enabling data. Currently individual agencies have to run their own spatial services but it would be much more efficient to have common spatial web services that all agencies could leverage.

Build data drive culture across APS

  1. Strong Recommendation: Embed data approaches are considered in all major government investments. For example, if data sensors were built into major infrastructure projects it would create more intelligence about how the infrastructure is used over time. If all major investments included data reporting then perhaps it would be easier to keep projects on time and budget.

  2. Recommendation: Establish a whole of government data skills program, not just for specialist skills, but to embed in the entire APS and understanding of data-driven approaches for the public service. This would ideally include mandatory data training for management (in the same way OH&S and procurement are mandatory training). At C is a draft approach that could be taken.

  3. Recommendation: Requirement that all government contracts have create new data make that data available to the contracting gov entity under Creative Commons By Attribution only licence so that government funded data is able to published publicly according to government policy. I have seen cases of contracts leaving ownership with companies and then the data not being reusable by government.

  4. Recommendation: Real data driven indicators required for all new policies, signed off by data champions group, with data for KPIs publicly available on data.gov.au for public access and to feed policy dashboards. Gov entities must identify existing data to feed KPIs where possible from gov, private sector, community and only propose new data collection where new data is clearly required.

  • Note: it was good to see a new requirement to include evidence based on data analytics for new policy proposals and to consult with the Data Champions about how data can support new proposals in the recently launched implementation report on the Public Data Management Report. However, I believe it needs to go further and require data driven indicators be identified up front and reported against throughout as per the recommendation above. Evidence to support a proposal does not necessarily provide the ongoing evidence to ensure implementation of the proposal is successful or has the intended effect, especially in a rapidly changing environment.

  1. Recommendation: Establish relationships with private sector to identify aggregate data points already used in private sector that could be leveraged by public sector rather. This would be more efficient and accurate then new data collection.

  2. Recommendation: Establish or extend a cross agency senior management data champions group with specific responsibilities to oversee the data agenda, sign off on data indicators for NPPs as realistic, provide advice to Government and Finance on data infrastructure proposals across the APS.

  3. Recommendation: Investigate the possibilities for improving or building data sharing environments for better sharing data between agencies.

  4. Recommendation: Take a distributed and federated approach to linking unit record data. Secure API access to sensitive data would avoid creating a honey pot.

  5. Recommendation: Establish data awards as part of annual ICT Awards to include: most innovative analytics, most useful data infrastructure, best data publisher, best data driven policy.

  6. Recommendation: Extend the whole of government service analytics capability started at the DTO and provide access to all agencies to tap into a whole of government view of how users interact with government services and websites. This function and intelligence, if developed as per the original vision, would provide critical evidence of user needs as well as the impact of changes and useful automated service performance metrics.

  7. Recommendation: Support data driven publishing including an XML platform for annual reports and budgets, a requirement for data underpinning all graphs and datavis in gov publications to be published on data.gov.au.

  8. Recommendation: develop a whole of government approach to unit record aggregation of sensitive data to get consistency of approach and aggregation.

Implementation recommendations

  1. Move the Public Data Branch to an implementation agency – Currently the Public Data Branch sits in the Department of Prime Minister and Cabinet. Considering this Department is a policy entity, the questions arises as to whether it is the right place in the longer term for an agenda which requires a strong implementation capability and focus. Public data infrastructure needs to be run like other whole of government infrastructure and would be better served as part of a broader online services delivery team. Possible options would include one of the shared services hubs, a data specialist agency with a whole of government mandate, or the office of the CTO (Finance) which runs a number of other whole of government services.

Downloadable copy

Finding the natural motivation for change

Update: I added a section on how competition can be motivating 🙂

I’ve had a lot of people and ideas in my life that have been useful to me so I wanted to share a theory I have applied in my work that might be useful to others. The concept of finding the ‘natural motivation’ of players involved is a key component when I’m planning any type of systemic change. This isn’t a particularly unique or new idea, but I am constantly surprised how rarely I see it adopted in practice, and how often things fail by not taking it into consideration. It is critical if you want to take a new idea from the domain of evangelists and into ‘business as usual’ because if you can’t embed something into the normal way people act and think, then whatever you are trying to do will be done reluctantly and at best, tacked on to normal processes as an afterthought.

In recent years I’ve been doing a lot of work to try to change systems, thinking and culture around open government, technology in government and open data, with some success. This is in part because I purposefully take an approach that tries to identify and tap into the natural motivation of all players involved. This means understanding how what I’m trying to do could benefit the very people who need to change their behaviours, and helping them want to do something new of their own volition. Why does this matter? If I asked you to spend an extra couple of hours a week at work, for no extra pay, doing something you don’t understand that seems completely unrelated to your job or life, you’d tell me to sod off. And understandably so! And yet we expect people and behaviours to simply comply if we change the rules. If I talked to you about how a new way of doing something would save you time, get a better outcome, save money or made life better in any way, you would be more interested. Then it simply becomes a matter of whether the effort is worth the benefit.

Die hard policy wonks will argue that you can always punish non compliance or create incentives if you are serious enough about the change you want to make. I would argue that you can force certain behaviour changes through punishment or reward, but if people aren’t naturally motivated to make the behaviour change themselves then the change will be both unsustainable and minimally implemented.

I’m going to use open data in government as my example of this in practice. Now I can hear a lot of people saying “well public servants should do open data by default because it is good for the community!” but remember the question above. In the first instance, if I’m asking someone to publish data without understanding why, they will see it as just extra work for no benefit – merely a compliance activity that gets in the way of their real work. People ask the understandable question of why would anyone want to divert resources and money into open data when it could be used to do something ‘real’ like build a road, deliver a better service, pay a salary, etc? Every day public servants are being asked to do more with less, so open data appears at first glance like a low priority. If the community and economy were to benefit from open data, then we had to figure out how to create a systemic change in government to publish open data naturally, or it would never scale or be sustainable.

When I took over data.gov.au, there was a reasonable number of datasets published but they weren’t being updated and nothing new was being added. It was a good first attempt, but open data had not really been normalised in agencies, so data publishing was sparodic. I quickly realised if open data was just seen as a policy and compliance issue, then this would never really change and we would hit a scaling issue of how much we could do ourselves. Through research, experimenting and experience, we did find that open data can help agencies be more effective, more efficient and more able to support an ecosystem of information and service delivery rather than all the pressure being on agencies to do everything. This was a relief because if there was no benefit to the public service itself, then realistically open data would always be prioritised lower than other activities, regardless of the political or policy whims of the few.

So we started working with agencies on the basis that although open data was the policy position that agencies were expected to adopt, there were real benefits to agencies if they adopted an open data approach. We would start an agency on the open data journey by helping them identify datasets that save them time and money, looking at resource intensive requests for data they regularly get and how to automate the publishing of that data. This then frees up resources of which a proportion can often be justified to start a small open data team. Whatever the agency motivations, there is always an opportunity for open data to support that goal if integrated properly. We focused on automation, building open data into existing processes (rather than creating a new process), supporting and promoting public reuse of data (GovHack was particularly helpful for this), identifying community priority datasets, raising public confidence in using government open data and removing barriers for publishing data. We knew centralised publishing would never scale, so we focused our efforts on a distributed publishing model where the central data.gov.au team provided technical support and a free platform for publishing data, but agencies did their own publishing with our help. Again this meant we had to help agencies understand how useful open data was to them so they could justify putting resources towards their own data publishing capacity. We knew agencies would need to report on their own success and progress with open data, so we also ensured they could access their own data utilisation analytics, which is also publicly available for a little extra motivation.

We collected examples from agecnies on the benefits to help inform and encourage other agencies, and found the key agency benefits of open data were broadly:

  1. Efficiency – proactively publishing data that is commonly asked for in an automated way frees up resources.
  2. Innovation – once data is published, so long as it is published well and kept up to date, other people and organisations will use the data to create new information, analysis and services. This innovation can be adopted by the agency, but it also takes the pressure off the agency to deliver all things to all people, by enabling others to scratch their own itch.
  3. Improved services – by publishing data in a programmatically accessible way, agencies found cheaper and more modular service delivery was possible through reusable data sources. Open data is often the first step for agencies on the path to more modular and API driven way of doing things (which the private sector embraced a decade ago). I believe if we could get government data, content and services API enabled by default, we would see dramatically cheaper and better services across all governments, with the opportunity for a public ecosystem of cross jurisdictional service and information delivery to emerge.

To extend the natural motivation consideration further, we realised that unless data was published in a way that people in the community could actually find and use, then all the publishing in the world would not help. We had to ensure the way data was publishing supported the natural motivation of people who want to use data, and this would in turn create a feedback loop to encourage greater publishing of data. We adopted a “civic hacker empathy” approach (with credit to Chris Gough for the concept) so that we always put ourselves in the shoes of those wanting to use data to prioritise how to publish it, and to inform and support agencies to publish data in a way that could be easily consumed. This meant agencies starting on the open data journey were not only encouraged to adopt good technical practices from day 1, but were clearly educated on the fact they wouldn’t yield the benefits from open data without publishing data well.

I should also mention that motivation doesn’t need to always come from within the individual person or the organisation. Sometimes motivation can come from a little healthy competition! I have had people in agencies utterly uninterested in open data that I’ve decided to not push (why spend effort on a closed door when there are partially open or open doors available!) who have become interested when other agencies have had some success. Don’t underestimate the power of public successes! Be as loud as you can about successes you have as this will build interest and demand, and help bring more people on your journey.

So to wrap up, I’ve been amazed how many people I meet, particularly in the federal government, who think they can change behaviour by simply having a policy, or law, or a financial incentive. The fact is, people will generally only do something because they want to, and this applies as much in the work place as anywhere else. If you try to force people to do something that they don’t want to, they will find myriad ways to avoid it or do the bare minimum they have to, which will never yield the best results. Every single barrier to open data we came across woud magically disappear if the agency and people involved were naturally morivated to do open data.

If you want to make real change, I encourage you to take an empathetic approach, think about all the players in the system, and how to ensure they are naturally motivated to change. I always tell the data.gov.au team that we always need to ensure the path of technical integrity is the path of least resistance, because this ensures an approach which is good for both the data publishers and data consumers. It goes without saying that a change is easiest to encourage when it has integrity and provides genuine benefits. In the case of open data, we simply needed to help others come on the journey for the idea to flourish. I’m proud to say the data.gov.au team have managed to dramatically increase the amount of open data available in Australia as well as support a rapidly growing capacity and appetite for open data throughout the public service. Huge kudos to the team! With the data.gov.au team now moved to the Department of Prime Minister and Cabinet, and merged with the spatial data branch from the Department of Communications, we have a stronger than ever central team to continue the journey.

Note: I should say I’m currently on maternity leave till the end of 2016, hence the time to publish some of these ideas. They are my own thoughts and not representative of any one else. I hope they are useful 🙂

Returning to data and Gov 2.0 from the DTO

I have been working at the newly created Digital Transformation Office in the Federal Government since January this year helping to set it up, create a vision, get some good people in and build some stuff. I was working in and then running a small, highly skilled and awesome team focused on how to dramatically improve information (websites) and transaction services across government. This included a bunch of cool ideas around whole of government service analytics, building a discovery layer (read APIs) for all government data, content and services, working with agencies to improve content and SEO, working on reporting mechanisms for the DTO, and looking at ways to usefully reduce the huge number of websites currently run by the Federal public service amongst other things. You can see some of our team blog posts about this work.

It has been an awesome trip and we built some great stuff, but now I need to return to my work on data, gov 2.0 and supporting the Australian Government CTO John Sheridan in looking at whole of government technology, procurement and common platforms. I can also work more closely with Sharyn Clarkson and the Online Services Branch on the range of whole of government platforms and solutions they run today, particularly the highly popular GovCMS. It has been a difficult choice but basically it came down to where my skills and efforts are best placed at this point in time. Plus I miss working on open data!

I wanted to say a final public thank you to everyone I worked with at the DTO, past and present. It has been a genuine privilege to work in the diverse teams and leadership from across over 20 agencies in the one team! It gave me a lot of insight to the different cultures, capabilities and assumptions in different departments, and I think we all challenged each other and created a bigger and better vision for the effort. I have learned much and enjoyed the collaborative nature of the broader DTO team.

I believe the DTO has two major opportunities ahead: as a a force of awesome and a catalyst for change. As a force of awesome, the DTO can show how delivery and service design can be done with modern tools and methods, can provide a safe sandpit for experimentation, can set the baseline for the whole APS through the digital service standard, and can support genuine culture change across the APS through training, guidance and provision of expertise/advisers in agencies. As a catalyst for change, the DTO can support the many, many people across the APS who want transformation, who want to do things better, and who can be further empowered, armed and supported to do just that through the work of the DTO. Building stronger relationships across the public services of Australia will be critical to this broader cultural change and evolution to modern technologies and methodologies.

I continue to support the efforts of the DTO and the broader digital transformation agenda and I wish Paul Shetler and the whole team good luck with an ambitious and inspiring vision for the future. If we could all make an approach that was data/evidence driven, user centric, mashable/modular, collaborative and cross government(s) the norm, we would overcome the natural silos of government, we would establish the truly collaborative public service we all crave and we would be better able to support the community. I have long believed that the path of technical integrity is the most important guiding principle of everything I do, and I will continue to contribute to the broader discussions about “digital transformation” in government.

Stay tuned for updates on the data.gov.au blog, and I look forward to spending the next 4 months kicking a few goals before I go on maternity leave 🙂

Embrace your inner geek: speech to launch QUT OSS community

This was a speech I gave in Brisbane to launch the QUT OSS group. It talks about FOSS, hacker culture, open government/data, and why we all need to embrace our inner geek 🙂

Welcome to the beginning of something magnificent. I have had the luck, privilege and honour to be involved in some pretty awesome things over the 15 or so years I’ve been in the tech sector, and I can honestly say it has been my involvement in the free and Open Source software community that has been one of the biggest contributors.

It has connected me to amazing and inspiring geeks and communities nationally and internationally, it has given me an appreciation of the fact that we are exactly as free as the tools we use and the skills we possess, it has given me a sense of great responsibility as part of the pioneer warrior class of our age, and it has given me the instincts and tools to do great things and route around issues that get in the way of awesomeness.

As such it is really excited to be part of launching this new student focused Open Source group at QUT, especially one with academic and industry backing so congratulations to QUT, Red Hat, Microsoft and Tech One.

It’s also worth mentioning that Open Source skills are in high demand, both nationally and internationally, and something like 2/3 of Open Source developers are doing so in some professional capacity.

So thanks in advance for having me, and I should say up front that I am here in a voluntary capacity and not to represent my employer or any other organisation.

Who am I? Many things: martial artist, musician, public servant, recently recovered ministerial adviser, but most of all, I am a proud and reasonably successful geek.

Geek Culture

So firstly, why does being a geek make me so proud? Because technology underpins everything we do in modern society. It underpins industry, progress, government, democracy, a more empowered, equitable and meritocratic society. Basically technology supports and enhances everything I care about, so being part of that sector means I can play some small part in making the world a better place.

It is the geeks of this world that create and forge the world we live in today. I like to go to non-geek events and tell people who usually take us completely for granted, “we made the Internet, you’re welcome”, just to try to embed a broader appreciation for tech literacy and creativity.

Geeks are the pioneers of the modern age. We are carving out the future one bit at a time, and leading the charge for mainstream culture. As such we have, I believe, a great responsibility to ensure our powers are used to improve life for all people, but that is another lecture entirely.

Geek culture is one of the driving forces of innovation and progress today, and it is organisations that embrace technology as an enabler and strategic benefit that are able to rapidly adapt to emerging opportunities and challenges.

FOSS culture is drawn very strongly from the hacker culture of the 60’s and 70’s. Unfortunately the term hacker has been stolen by the media and spooks to imply bad or illegal behaviours, which we would refer to as black hat hacking or cracking. But true hacker culture is all about being creative and clever with technology, building cool stuff, showing off one’s skills, scratching an itch.

Hacker culture led to free software culture in the 80’s and 90’s, also known as Open Source in business speak, which also led to a broader free culture movement in the 90’s and 00’s with Creative Commons, Wikipedia and other online cultural commons. And now we are seeing a strong emergence of open government and open science movements which is very exciting.

Open Source

A lot of people are aware of the enormity of Wikipedia. Even though Open Source well predates Wikipedia, it ends up being a good tool to articulate to the general population the importance of Open Source.

Wikipedia is a globally crowdsourced phenomenon than, love it or hate it, has made knowledge more accessible than every before. I personally believe that the greatest success of Wikipedia is in demonstrating that truth is perception, and the “truth” held in the pages of Wikipedia ends up, ideally anyway, being the most credible middle ground of perspectives available. The discussion pages of any page give a wonderful insight to any contradicting perspectives or controversies and it teaches us the importance of taking everything with a grain of salt.

Open Source is the software equivalent of Wikipedia. There are literally hundreds of thousands if not millions of Open Source software projects in the world, and you would used thousands of the most mature and useful ones every day, without even knowing it. Open Source operating systems like Linux or MINIX powers your cars, devices, phones, telephone exchanges and the majority of servers and super computers in the world. Open Source web tools like WordPress, Drupal or indeed WikiMedia (the software behind Wikipedia) power an enormous amount of websites you go to everyday. Even Google heavily uses Open Source software to build the worlds most reliable infrastructure. If Google.com doesn’t work, you generally check your own network reliability first.

Open Source is all about people working together to scratch a mutual itch, sharing in the development and maintenance of software that is developed in an open and collaborative way. You can build on the top of existing Open Source software platforms as a technical foundation for innovation, or employ Open Source development methodologies to better innovate internally. I’m still terrified by the number of organisations I see that don’t use base code revision systems and email around zip files!

Open Source means you can leverage expertise far beyond what you could ever hope to hire, and you build your business around services. The IT sector used to be all about services before the proprietary lowest common denominator approach to software emerged in the 80s.

But we have seen the IT sector largely swing heavily back to services, except in the case on niche software markets, and companies compete on quality of services and whole solution delivery rather than specific products. Services companies that leverage Open Source often find their cost of delivery lower, particularly in the age of “cloud” software as a service, where customers want to access software functionality as a utility based on usage.

Open Source can help improve quality and cost effectiveness of technology solutions as it creates greater competition at the services level.

The Open Source movement has given us an enormous collective repository of stable, useful, innovative, responsive and secure software solutions. I must emphasise secure because many eyes reviewing code means a better chance of identifying and fixing issues. Security through obscurity is a myth and it always frustrates me when people buy into the line that Open Source is somehow less secure than proprietary solutions because you can see the code.

If you want to know about government use of Open Source, check out the Open Source policy on the Department of Finance and Deregulation website. It’s a pretty good policy not only because it encourages procurement processes to consider Open Source equally, but because it encourages government agencies to contribute to and get involved in the Open Source community.

Open Government

It has been fascinating to see a lot of Open Source geeks taking their instincts and skills with them into other avenues. And to see non-technical and non-Open Source people converging on the same basic principles of openness and collaboration for mutual gain from completely different avenues.

For me, the most exciting recent evolution of hacker ethos is the Open Government movement.

Open Government has always been associated with parliamentary and bureacratic transparency bureaucratic, such as Freedom of Information and Hansard.

I currently work primarily on the nexus where open government meets technology. Where we start to look at what government means in a digital age where citizens are more empowered than ever before, where globalisation challenges sovereignty, where the need to adapt and evolve in the public service is vital to provide iterative, personalised and timely responses to new challenges and opportunities both locally and globally.

There are three key pillars of what we like to call “Government 2.0”. A stupid term I know, but bear with me:

  1. Participatory governance – this is about engaging the broader public in the decision making processes of government to both leverage the skills, expertise and knowledge of the population for better policy outcomes, and to give citizens a way to engage directly with decisions and programs that affect their every day lives. Many people think about democratic engagement as political engagement, but I content that the public service has a big role to play in engaging citizens directly in co-developing the future together.
  2. Citizen centricity – this is about designing government services with the citizen at the centre of the design. Imagine if you will, and I know many in the room are somewhat technical, imagine government as an API, where you can easily aggregate information and services thematically or in a deeply personalised way for citizens, regardless of the structure or machinery of government changes. Imagine being able to change your address in one location, and have one place to ask questions or get the services you need. This is the vision of my.gov.au and indeed there are several initiatives that delivery on this vision including the Canberra Connect service in the ACT, which is worth looking at. In the ACT you can go into any Canberra Connect location for all your Territory/Local government needs, and they then interface with all the systems of that government behind the scenes in a way that is seamless to a citizen. It is vital that governments and agencies start to realise that citizens don’t care about the structures of government, and neither should they have to. It is up to us all to start thinking about how we do government in a whole of government way to best serve the public.
  3. Open and transparent government – this translates as both parliamentary transparency, but also opening up government data and APIs. Open data also opens up opportunities for greater analysis, policy development, mobile service delivery, public transaprency and trust, economic development through new services and products being developed in the private sector, and much more.

Open Data

Open data is very much my personal focus at the moment. I’m now in charge of data.gov.au, which we are in the process of migrating to an excellent Open Source data repository called CKAN which will be up soon. There is currently a beta up for people to play with.

I also am the head cat herder for a volunteer run project called GovHack which ran only just a week ago, where we had 1000 participants from 8 cities, including here in Brisbane, all working with government data to build 130 new hacks including mashups, data visualisations, mobile and other applications, interactive websites and more. GovHack shows clearly the benefits to society when you open up government data for public use, particularly if it is available in a machine readable way and is available under a very permissive copyright such as Creative Commons.

I would highly recommend you check out my blog posts about open data around the world from when I went to a conference in Helsinki last year and got to meet luminaries in this space including Hans Rosling, Dr Tim Hubbard and Rufus Pollock. I also did some work with the New Zealand Government looking at NZ open data practice and policy which might be useful, where we were also able to identify some major imperatives for changing how governments work.

The exciting thing is how keen government agencies in Federal, State, Territory and Local governments are to open up their data! To engage meaningfully with citizens. And to evolve their service delivery to be more personalised and effective for everyone. We are truly living in a very exciting time for technologists, democracy and the broader society.

Though to be fair, governments don’t really have much choice. Citizens are more empowered than ever before and governments have to adapt, delivery responsive, iterative and personalised services and policy, or risk losing relevance. We have seen the massive distribution now of every traditional bastion of power, from publishing, communications, monitoring, enforcement, and even property is about to dramatically shift, with the leaps in 3D printing and nano technologies. Ultimately governments are under a lot of pressure to adapt the way we do things, and it is a wonderful thing.

The Federal Australian Government already has in place several policies that directly support opening up government data:

Australia has also recently signed up to the Open Government Partnership, an international consortia of over 65 governments which will be a very exciting step for open data and other aspects of open government.

At the State and Territory level, there is also a lot of movement around open data. Queensland and the ACT launched your new open data platform late last year with some good success. NSW and South Australia have launched new platforms in the last few weeks with hundreds of new data sets. Western Australia and Victoria have been publishing some great data for some time and everyone is looking at how they can do so better!

Many local governments have been very active in trying to open up data, and a huge shout out to the Gold Coast City Council here in Queensland who have been working very hard and doing great things in this space!

It is worth noting that the NSW government currently have a big open data policy consultation happening which closes on the 17th June and is well worth looking into and contributing to.

Embracing geekiness

One of my biggest bug bears is when people say “I’m sorry the software can’t do that”. It is the learned helplessness of the tech illiterate that is our biggest challenge for innovating and being globally competitive, and as countries like Australia are overwhelming well off, with the vast majority of our citizens living high quality lives, it is this learned helplessness that is becoming the difference between the haves and have nots. The empowered and the disempowered.

Teaching everyone to embrace their inner geek isn’t just about improving productivity, efficiency, innovation and competitiveness, it is about empowering our people to be safer, smarter, more collaborative and more empowered citizens in a digital world.

If everyone learnt and experienced even the tiniest amount of programming, we would all have embedded that wonderful instinct that says “the software can do whatever we can imagine”.

Open Source communities and ethos gives us a clear vision as to how we can overcome every traditional barrier to collaboration to make awesome stuff in a sustainable way. It teaches us that enlightened self interest in the age of the Internet translates directly to open and mutually beneficial collaboration.

We can all stand on the shoulders of giants that have come before, and become the giants that support the next generation of pioneers. We can all contribute to making this world just a bit more awesome.

So get out there, embrace your inner geek and join the open movement. Be it Open Source, open government or open knowledge, and whatever your particular skills, you can help shape the future for us all.

Thank you for coming today, thank you to Jim for inviting me to be a part of this launch, and good luck to you all in your endeavours with this new project. I look forward to working with you to create the future of our society, together.

My NZ Open Data and Digital Government Adventure

On a recent trip to New Zealand I spent three action packed days working with Keitha Booth and Alison Stringer looking at open data. These two have an incredible amount of knowledge and experience to share, and it was an absolute pleasure to work with them, albeit briefly. They arranged meetings with about 3000* individuals from across different parts of the NZ government to talk about everything from open data, ICT policy, the role of government in a digital era, iterative policy, public engagement and the components that make up a feasible strategy for all of the above.

It’s important to note, I did this trip in a personal capacity only, and was sure to be clear I was not representing the Australian government in any official sense. I saw it as a bit of a public servant cultural exchange, which I think is probably a good idea even between agencies let alone governments 😉

I got to hear about some of the key NZ Government data projects, including data.govt.nz, data.linz.govt.nz, the statistical data service, some additional geospatial and linked data work, some NZ government planning and efforts around innovation and finding more efficient ways to do tech, and much more. I also found myself in various conversations with extremely clever people about science and government communications, public engagement, rockets, circus and more.

It was awesome, inspiring, informative and exhausting. But this blog post aims to capture the key ideas from the visit. I’d love your feedback on the ideas/frameworks below, and I’ll extrapolate on some of these ideas in followup posts.

I’m also looking forward to working more collaboratively with my colleagues in New Zealand, as well as from across all three spheres of government in Australia. I’d like to set up a way for government people in the open data and open government space across Australia/New Zealand to freely share information and technologies (in code), identify opportunities to collaborate, share their policies and planning for feedback and ideas, and generally work together for more awesome outcomes all round. Any suggestions for how best to do this? 🙂 GovDex? A new thing? Will continue public discussions on the Gov 2.0 mailing list, but I think it’ll be also useful to connect govvies privately whilst encouraging individuals and agencies to promote their work publicly.

This blog post is a collaboration with the wonderful Alison Stringer, in a personal capacity only. Enjoy!

* 3000 may be a wee stretch 🙂

Table of Contents

Open Data

  • Strategic/Policy Building Blocks
  • Technical Building Blocks
  • References

Digital and Open Government

  • Some imperatives for changing how we do government
  • Policy/strategic components

Open data

Strategic/policy building blocks

Below are some basic building blocks we have found to be needed for an open data strategy to be sustainable and effective in gaining value for both the government and the broader community including industry, academia and civil society. It is based on the experiences in NZ, Aus and discussions with open data colleagues around the world. Would love your feedback, and I’ll expand this out to a broader post in the coming weeks.

  • Policy– open as the default, specifically encouraging and supporting a proactive and automated disclosure of government information in an appropriate, secure and sustainable way. Ideally, each policy should be managed as an iterative and live document that responds to changing trends, opportunities and challenges:
    • Copyright and licensing – providing clear guidance that government information can be legally used. Using simple, permissive and known/trusted licences is important to avoid confusion.
    • Procurement – procurement policy creates a useful and efficient lever to establish proactive “business as usual” disclosure of information assets, by requiring new systems to support such functionality and publishing in open data formats from the start. This also means the security and privacy of data can be built into the system.
    • Proactive publishing – a policy of proactive disclosure helps avoid the inefficiencies of retrospective data publishing. It is important to also review existing assets and require an implementation plan from all parts of government on how they will open up their information assets, and then measure, monitor and report on the progress.
  • Legislation – ensuring any legislative blockers to publishing data are sorted, for instance, in some jurisdictions civil servants are personally liable if someone takes umbrage to the publication of something. Indeed there may be some issues here that are perceptions as opposed to reality. A review of any relevant legislation and plan to fix any blockers to publishing information assets is recommended.
  • Leadership/permission – this is vital, especially in early days whilst open data is still being integrated as business as usual. It should be as senior as possible.
  • Resourcing – it is very hard to find new money in governments in the current fiscal environment. However, we do have people. Resourcing the technical aspects of an open data project would only need a couple of people and a little infrastructure that can both host and point to data and data services. The UK open data platform runs on less than £460K per year, including the costs of three staff). But there needs to be a policy of distributed publishing. In the UK there are ~760 registered publishers of data throughout government. It would be useful to have at least one data publisher (probably to work part of their job only and alongside the current senior agency data champion role) who spends a day or two a week just seeking out and publishing data for their department, and identifying opportunities to automate data publishing with the data.govt.nz team.
  • Value realisation – including:
    • Improved policy development across government through better and early access to data and tools to use data
    • Knowledge transfer across government, especially given so many senior public servants are retiring in the coming years
    • Improved communication of complex issues to the public, better public engagement and exploration of data – especially with data visualisation tools
    • Monitoring, reporting, measuring clear outcomes (productivity savings, commercialisation, new business or products/projects, innovation in government, improved efficiency in Freedom of Information responses, efficiencies in not replicating data or reports, effectiveness and metrics around projects, programs and portfolios)
    • Application of data in developing citizen centric services and information
    • Supporting and facilitating commercialisation opportunities
  • Agency collaboration – the importance of agency collaboration can not be overstated. Especially on sharing/using/reusing data, on sharing knowledge and skills, on public engagement and communications. Also on working together where projects or policy areas might be mutually beneficial and on public engagement such that there is a consistent and effective dialogue with citizens. This shouldn’t be a bottlenecked approach, but rather a distributed network of individuals at different levels and in different functions.
  • Technology – need to have the right bits in place, or the best policy/vision won’t go anywhere 🙂 See below for an extrapolation on the technical building blocks.
  • Public engagement – a public communications and engagement strategy is vital to build and support a community of interest and innovation around government data.

Technical building blocks

Below are some potential technical building blocks for supporting a whole of government(s) approach to information management, proactive publishing and collaboration. Let me know what you think I’m missing 🙂

Please note, I am not in any way suggesting this should be a functional scope for a single tool. On the contrary, I would suggest for each functional requirement the best of breed tool be found and that there be a modular approach such that you can replace components as they are upgraded or as better alternatives arise. There is no reason why a clever frontend tool couldn’t talk to a number of backend services.

  • Copyright and licensing management – if an appropriately permissive copyright licence is applied to data/content at the point of creation, and stored in the metadata, it saves on the cost of administration down the track. The Australian Government default license has been determined as Creative Commons BY, so agencies and departments should use that, regardless of whether the data/content is ever publishing publicly. The New Zealand government recommends CC-BY as the default for data and information published for re-use.
  • An effective data publishing platform(s) (see Craig Thomler’s useful postabout different generations of open data platforms) that supports the publishing, indexing and federation of data sources/services including:
    • Geospatial data – one of the pivotal data sets required for achieving citizen centric services, and in bringing the various other datasets together for analysis and policy development.
    • Real time data – eg, buses, weather, sensor networks
    • Statistical data – eg census and surveys, where raw access to data is only possible through an API that gives a minimum number of results so as to make individual identification difficult
    • Tabular data – such as spreadsheets or databases of records in structured format
  • Identity management – for publishers at the very least.
  • Linked data and metadata system(s) – particularly where such data can be automatically inferred or drawn from other systems.
  • Change control – the ability to push or take updates to datasets, or multiple files in a dataset, including iterative updates from public or private sources in a verifiable way.
  • Automation tools for publishing and updating datasets including where possible, from their source, proactive system-to-system publishing.
  • Data analysis and visualisation tools – both to make it easier to communicate data, but also to help people (in government and the public) analyse and interact with any number of published datasets more effectively. This is far more efficient for government than each department trying to source their own data visualisation and analysis tools.
  • Reporting tools – that clearly demonstrate status, progress, trends and value of open data and open government on an ongoing basis. Ideally this would also feed into a governance process to iteratively improve the relevant policies on an ongoing basis.

Some open data references

Digital and Open Government

Although I was primarily in New Zealand to discuss open data, I ended up entering into a number of discussions about the broader aspects of digital and open government, which is entirely appropriate and a natural evolution. I was reminded of the three pillars of open government that we often discuss in Australia which roughly translate to:

  • Transparency
  • Participation
  • Citizen centricity

There is a good speech by my old boss, Minister Kate Lundy, which explains these in some detail.

I got into a couple of discussions which went into the concept of public engagement at length. I highly recommend those people check out the Public Sphere consultation methodology that I developed with Minister Kate Lundy which is purposefully modular so that you can adapt it to any community and how they best communicate, digitally or otherwise. It also is focused on getting evidence based, peer reviewed, contextually analysed and useful actual outcomes. It got an international award from the World eDemocracy Forum, which was great to see. Particularly check out how we applied computer forensics tools to help figure out if a consultation is being gamed by any individual or group.

When I consider digital government, I find myself standing back in the first instance to consider the general role of government in a digital society. I think this is an important starting point as our understanding is broadly out of date. New Zealand has definitions in the State Sector Act 1988, but they aren’t necessarily very relevant to 2013, let alone an open and transparent digital government.

Some imperatives for changing how we do government

Below are some of the interesting imperatives I have identified as key drivers for changing how we do government:

  • Changing public expectations – public expectations have fundamentally changed, not just with technology and everyone being connected to each other via ubiquitous mobile computing, but our basic assumptions and instincts are changing, such as the innate assumption of routing around damage, where damage might be technical or social. I’ve gone into my observations in some depth in a blog post called Online Culture – Part 1: Unicorns and Doom (2011).
  • Tipping point of digital engagement with government – in 2009 Australia had more citizens engaging with government  online than through any other means. This digital tipping point creates a strong business case to move to digitally delivered services, as a digital approach enables more citizens to self serve online and frees up expensive human resources for our more vulnerable, complex or disengaged members of the community.
  • Fiscal constraints over a number of years have largely led to IT Departments having done more for less for years, with limited investment in doing things differently, and effectively a legacy technology millstone. New investment is needed but no one has money for it, and IT Departments have in many cases, resorted to being focused on maintenance rather than project work (an upgrade of a system that maintains the status quo is still maintenance in my books). Systems have reached a difficult point where the fat has been trimmed and trimmed, but the demands have grown. In order to scale government services to growing needs in a way that enables more citizens to self service, new approaches are necessary, and the capability to aggregate services and information (through open APIs and open data) as well as user-centric design underpins this capability.
  • Disconnect between business and IT – there has been for some time a growing problem of business units disengaging with IT. As cheap cloud services have started to appear, many parts of government (esp Comms and HR) have more recently started to just avoid IT altogether and do their own thing. On one hand this enables some more innovative approaches, but it also leads directly to a problem in whole of government consistency, reliability, standards and generally a distribution of services which is the exact opposite of a citizen centric approach. It’s important that we figure out how to get IT re-engaged in the business, policy and strategic development of government such that these approaches are more informed and implementable, and such that governments use, develop, fund and prioritise technology in alignment with a broader vision.
  • Highly connected and mobile community and workforce – the opportunities (and risks) are immense, and it is important that governments take an informed and sustainable approach to this space. For instance, in developing public facing mobile services, a mobile optimised web services approach is more inclusive, cost efficient and sustainable than native applications development, but by making secure system APIs and open data available, the government can also facilitate public and private competition and innovation in services delivery.
  • New opportunities for high speed Internet are obviously a big deal in Australia and New Zealand at the moment with the new infrastructure being rolled out (FTTP in both countries), and setting up to better support and engaging with citizens digitally now, before mainstream adoption, is rather important and urgent.
  • Impact of politics and media on policy – the public service is generally to have an evidence-based approach to policy, and where this approach is developed in a transparent and iterative way, in collaboration with the broader society, it means government can engage directly with citizens rather than through the prism of politics or the media, each which have their own motivations and imperatives.
  • Prioritisation of ICT spending – it is difficult to ensure the government investment and prioritisation of ICT projects aligns with the strategic goals of the organisation and government, especially where the goals are not clearly articulated.
  • Communications and value realisation – with anyone able to publish pretty much anything, it is incumbent on governments to be a part of the public narrative as custodians of a lot of information and research. By doing this in a transparent and apolitical way, the public service can be a value and trusted source.
  • The expensive overhead of replication of effort across governments – consolidating where possible is vital to improve efficiencies, but also to put in place the mechanisms to support whole of government approaches.
  • Skills – a high technical literacy directly supports the capacity to innovate across government and across the society in every sector. As such this should be prioritised in our education systems, way above and well beyond “office productivity” tools.

Policy/strategic components

  • Strategic approach to information policy – many people looking at information policy tend to look deeply at one or a small number of areas, but it is only in looking at all of the information created by government, and how we can share, link, re-use, and analyse that we will gain the significant policy, service delivery and social/economic benefits and opportunities. When one considers geospatial, tabular, real time and statistical (census and survey) data, and then the application of metadata and linked data, it gets rather complicated. But we need to be able to interface effectively with these different data types.
  • Facilitating public and private innovation – taking a “government as a platform” approach, including open data and open APIs, such that industry and civil society can innovate on top of government systems and information assets, creating new value and services to the community.
  • Sector and R&D investment – it is vital that government ensured that the investment in digital industries, internal innovation and indeed R&D more broadly, aligns with the strategic vision. This means understanding how to measure and monitor digital innovation more effectively and not through the lens of traditional approaches that may not be relevant, such as the number of patents and other IP metrics. The New Zealand and Australian business and research community need to make the most of their governments’ leadership in Open Government. The Open Government Partnership network might provide a way to build upon and export this expertise.
  • Exports – by creating local capacity in the arena of improved and citizen-centric services delivery, Australia and New Zealand set themselves up nicely for exporting services and products to Asia Pacific, particularly given the rapid uptake of countries in the region to join the Open Government Partnership which requires signatories to develop plans around topics such as open data, citizen centricity and parliamentary transparency, all of which we are quite skilled in.
  • Distributed skunkworks for government – developing the communities/spaces/tools across government to encourage and leverage the skills and enthusiasm of clever geeks both internally (internal hackdays, communities of practice) and externally (eg – GovHack). No one can afford new resources, but allocating a small amount of time from the existing workforce who are motivated to do great things is a cost efficient and empowering way to create a distributed skunkworks. And as people speak to each other about common problems and common solutions we should see less duplication of these solutions and improved efficiency across agencies.
  • Iterative policy – rethinking how policy is developed, implemented, measured and governed to take a more iterative and agile approach that a) leverages the skills and expertise of the broader community for more evidence based and peer reviewed policy outcomes and b) is capable of responding effectively and in a timely manner to new challenges and opportunities as they arise. It would also be useful to build better internal intelligence systems for an improved understanding of the status of projects, and improved strategic planning for success.
  • An Information Commissioner for New Zealand – an option for a policy lead on information management to work closely with departments to have a consolidated, consistent, effective and overall strategic approach to the management, sharing and benefits realisation of government information. This would also build the profile of Open Government in New Zealand and hopefully be the permanent solution to current resourcing challenges. The Office of the Australian Information Commissioner, and similar roles at State level, include the function of Information Commissioner, Privacy Commissioner and Free of Information Commissioner, and these combined give a holistic approach to government information policy that ideally balances open information and privacy. In New Zealand it could be a role that builds on recent information policies, such as NZGOAL which is designed, amongst other things, to replace bespoke content licences. Bespoke licences create an unnecessary liability issue for departments.
  • Citizen centricity – the increasing importance of consolidating government service and information delivery, putting citizens (and business) at the centre of the design. This is achieved through open mechanisms (eg, APIs) to interface with government systems and information such that they can be managed in a distributed and secure way, but aggregated in a thematic way.
  • Shared infrastructure and services – the shared services being taken up by some parts of the New Zealand Government is very encouraging to see, particularly when such an approach has been very successful in the ACT and SA state governments in Australia, and with several shared infrastructure and services projects at a national level in Australia including the AGIMO network and online services, and the NECTAR examples (free cloud stack tools for researchers). Shared services create the capacity for a consistent and consolidated approach, as well as enable the foundations of citizen centric design in a practical sense.

Some additional reading and thoughts

Digital literacy and ICT skills – should be embedded into curriculum and encouraged across the board. I did a paper on this as a contribution to the National Australian Curriculum consultation in 2010 with Senator Kate Lundy which identified three areas of ICT competency: 1) Productivity skills, 2) Online engagement skills, & 3) Automation skills as key skills for all citizens. It’s also worth looking at the NSW Digital Citizenship courseware. It’s worth noting that public libraries are a low cost and effective way to deliver digital services, information and skills to the broader community and minimise the issue of the digital divide.

Media data – often when talking about open data, media is completely forgotten. Video, audio, arts, etc. The GLAM (galleries, libraries, archives and museums) are all over this and should be part of the conversation about how to manage this kind of content across whole of government.

Just a few additional links for those interested, somewhat related to some of the things I discussed this last week.

OKFestival 2012: Open Data, Open Gov & Open Science in Helsinki

A couple of weeks ago I went to Helsinki, Finland to attend OKFestival 2012. It was a suggestion from someone two months ago that planted the seed to go, and I felt it would be really useful. So I saved my pennies and booked the tickets. It was an incredible trip with some incredible learnings.

Check out my “storify” stories which collate my experiences from the two days from my live Tweeting.

Rosling Hubbard Pollock & WaughHanging out with Hans Rosling, Rufus Pollock and Dr Tim Hubbard 🙂

Basically I’ve been working on open government and open data policy and projects for a while now and I realised I had a good opportunity to connect with practitioners and policy makers around the world. I really wanted to pick the brains of these people and also share what is happening locally to share, and to get some context on how we are going in a global context.

OKFestival name badgeI found many surprising things, not least of all that we are actually comparatively quite well in Australia when you look around the world. Obviously we have a lot to learn and do but we are lucky in many respects, such as we have a relatively open democracy already. For instance, Hansard for Federal and State parliamentary reporting is far from perfect but many countries have abysmal parliamentary openness and transparency. It was quite a shock to realise how little the accountability some jurisdictions operate under. More on that later.

It was also very interesting to hear from over 20 countries on their open data initiatives, to attend technical sessions on publishing data, to hear about the European Commission investment in open data (substantial!), and to talk to people from over 15 countries involved in “apps contests” and other hackfests. There was a lot of interest in our GovHack model so there might be some grounds for collaboration there too.

Statue in Helsinki with birdsI should also note that upon careful consideration, I thought it might be useful to bootstrap an OKFN Au local chapter in order to pull together all the open knowledge communities across Australia. Some network mechanism is needed as we have growing communities that are completely disconnected from one another. We could all benefit from some cross-disciplinary community development that includes cross promotion, discussions, aggregated events and news, tools for collaboration, support mechanisms (financial, insurance, legal, etc) and perhaps some events that bring us all together for mutual benefit.

So, this is my mother-of-all-posts report from the week. I will be blogging on some of the thoughts that have coalesced as a result later, but check out some of my highlights from the week below along with some really useful links. I’m also going to be working with the open government community people at OKFN to do an expanded open data census that looks at specific details of open data initiatives around the world to identify some good practice, policy commonalities and general information for people trying to do open data in government.

Open Government

The Open Government Partnership

Hanging out with Richard Akerman from Canada (@scilib)
Hanging out with Richard Akerman from Canada (@scilib)

The Open Government Partnership was a key theme for the conference, with over 55 countries now signed up in its first year. Signing up is not only a statement of commitment to this area, but countries have a series of targets on openness and transparency to meet. Apparently OGP has been slower to take off in Asia and Oceania, with only a few countries in this region getting involved to date.

Australia is unfortunately not yet signed up, and I hope that is rectified soon so Australia can more legitimately take our place in this space as something of an emerging leader. I had a lot of people interested in what Australia is doing at the conference from jurisdictions all around the world, and yet whenever we got to OGP discussions, there was not official Australian voice or commitment, which was disappointing. I hope this can be rectified soon, especially as the OGP commitments are already in line with so many existing policies in Australia.

Check out the infographic on the first year of progress of the OGP, and the draft strategic plan which is currently open for public comment.

There were several sessions on the OGP talking about standards, implementation challenges, and many representatives from supporting organisations like the World Bank who are investing in open government initiatives around the world.

Declaration of Parliamentary Openness

Afternoon tea with @anked & @kate_Braybrooke
Afternoon tea with @anked & @kat_Braybrooke

The Declaration of Parliamentary Openness was launched quite recently as an outcome of a global meeting of parliamentary monitoring organisations (PMOs). It is quite an interesting document and again, possibly something Australia should consider signing up to.

NSW Member of Parliament the Hon. Penny Sharpe did a great speech on the Declaration of Parliamentary Openness for International Day of Democracy (September 15th) which happened to be whilst I was in Helsinki. Several people there were very excited about the speech and I was quite honoured to be cited in it 🙂 Nice work Penny!

Check out some of the work from open parliaments around the world.

Open Data around the world

Gorgeous model in the basement of the uni
Gorgeous model in the basement of the uni

I managed to have a long sit down with the technical lead on data.gov.uk which was fantastic! It was great to get an idea of the model they use for publishing, the development work they have done, what resources they have an more.

My notes on the data.gov.uk discussion, with permission from their technical lead:

  • Human resources for data.gov.uk – 3  full time resources only
  • Uses CKAN – very happy with it, especially as they can easily develop additional functionality they need
  • Every department and local authority has at least one data champion that does data publishing as part of their normal job, ~765 publishers
  • Total cost of data.gov.uk only about 460k pound per year. 40k pound hosting and staff = most of the rest
  • Primarily focused on publishing data in the best way possible. Not focused on datavis, but considering looking at drupal front end with ckan backend
  • Departments are entirely responsibility for publishing their data. The full time staff look after the platform, do development where necessary (have created several plugins specific to their needs and open sourced them), provide technical support to publishers, but onus is on publishers
  • data.gov.uk folk have built functionality to handle the structure of government, creating lists of “Publishers” which are individual agencies (etc), users have a list of what Publishers they have access to publish to. You can have hierarchies of Publishers to reflect interrelationships between Publishers
  • An account API which could be the corporate API. Only publishers get API keys
  • No token required for apps
  • Antonio gave us a demonstration of uploading datasets, uploading had an option to choose whether a dataset is part of a time set
  • All datasets are appended, content is not changed at all, “if you get into data changing you are dead”
  • 5 star rating is helping improve quality of data publishing
  • With a multiple data file time series, the API interrogates the entire set
  • Contact details are available by dataset
  • data.gov.uk do thematic theming, they have over 8000 tags in the system atm, and they created 6 themes: health, environment, education, finance (other things apart from spending), society, defence, transportation, spending data (where they spend money), government
  • Automatic updates for some files via JSON but largely manual. Publishers felt more comfortable with manual publishing than aautomation for perceived control
  • Tend to point to WMS servers for spatial data rather than host directly
  • UK folk suggest a geoserver to host geospatial data and use open data platform to point to data rather than host it directly. A metadata harvester gets data from spatial sets and points to data. Needed to comply with the INSPIRE directive
  • They don’t apply 5 star to mapped data (or other purely linked information) as it doesn’t exactly map to downloadable data star rating
  • You can search on geospatial datasets by postcode or by drawing an area
  • Found that within a minute and 15 seconds (the record) a user could go from not having used the site before to publishing data, very low transition from newbie to publisher which was important
  • All statistics are automated which is due to being within the one dept and they are motivated to automated
  • INSPIRE (Infrastructure for Spatial Information in the European Community) Directive was a major driver, as was “digital by default”
  • They generate monthly reports that counts the openness (stars) of data, the amount per Publisher, publishers with broken links, datasets with broken links. Helps publishers keep their data up to date
  • data.gov.uk is building a dashboard to report by the hierarchy of government
  • Public Roles and Salaries linked data tool – http://data.gov.uk/organogram/cabinet-office
  • Blog post about plugins data.gov.uk have built, all freely available on github – http://data.gov.uk/blog/the-code-behind-datagovuk
  • Indemnification from the Crown so public servants not at personal legal risk
  • Started with the knowledge management people, then expanded. Basically all parts of the public service were told that this is what they must do, so they did it
  • data.gov.uk is hosted by the government, Ubuntu servers
  • data.gov.uk – metadata, almost a petabyte of data now
  • US is running three open data platforms, including Socrata, CKAN and another bespoke one
  • No inferred metadata – up to data publishers to provide metadata
  • Real time data – can deal with real time, new functionality being also built
  • The value of cloud service to scale with API requirements

For some technical details and the code behind data.gov.uk, check out http://data.gov.uk/blog/the-code-behind-datagovuk

Thanks Toni for your time!

Below are my live tweets on the open data country updates – each person had about 5 minutes to wrap up their country. I’ve put them in alphabetical order and the results were a fascinating snapshot. I wish I’d had more time to talk to each and every contributor:

  • Argentina – created Ministry of Modernisation inc Buenos Aires Data, 3 hackathons, datavis, app comp, digital city event coming
  • Australia – @piawaugh giving Australia report http://twitpic.com/avwd85
  • Australia – What’s going on in#opendata in #australia ? Psi needs to be cc by default. http://instagr.am/p/Ptj4FUodS9/ by Lucy Chambers
  • Australia – Not OGP members, national picture mixed, neat local efforts #okfest http://pic.twitter.com/l9u1JK1l by Tariq Khokar
  • Belgium – some progress, inconsistent across region. Estonia: need to transform data that is published to use as hard atm.
  • Brasil – have also done information asset catalogue to help facilitate future opendata.
  • Brasil – new law created to get important datasets published, also have proactive publishing of source code. Cost seen as blocker
  • Canada – over 12k datasets this year. Next gen platform deployed next year. Toronto building a city that thinks like the web 🙂
  • Canada – lots of municipal level work, national level is participant in OGP with three pillars of opendata openinfo & opendialogue
  • Chile – regulation around #opendata created but not implemented yet.
  • Czech – working on apps&services based on opendata, OKFN local chapter, data catalogue & opendata.cz, 1st gov data blog. No money
  • France – France presentation a reminder that open standards can actually be a blocker to first steps in opening data. #okfest
  • France – talk from NosDeputes.fr, lots of cities putting data up, national now commit to open licence, formats an issue/blocker
  • Germany case study of getting gov to open up some data, but really just getting stated.
  • Ireland – new real time passenger info API coming this year, need national portal, still low priority for many but relatively cheap
  • Ireland – 8-9 public bodies in Dublin regional opendata portal. Interested in biz models, datavis, 40% participants entrepreneurs
  • Israel – black whole of legislation, printing protocols were hid in boxes. Volunteers went in to scan and digitise. Now gov opened
  • Italy – National data portal, people need gov to open data, 3000 datasets liberati 🙂 increase in data quality. Mostly in north
  • Kenya – had one year birthday for Kenya opendata portal, focus on open standards, lots happening. No FOI leg yet, community devel
  • Netherlands – parliamentary data opened, 1st budget open data tmrw, issues: budget cuts hard, slow grow, gov benefit realisation
  • Netherlands – launched open data portal, gov stopped charging for geospatial data, $4m spent to free up satellite images…
  • Nigeria – update data hard to get, oil companies & gov corruption high, have digitised & visualised budget -> public engagement
  • Open Corporates – has info on 44,470,772 companies in the world. Open database of the corporate world. Interesting. Launched API
  • Slovakia – bad news is a lot of new laws but the working group works are slow and projects at risk.
  • Slovakia – launched open data portal, has preliminary support, worried about new gov not supporting but did, slow but building up
  • South Africa – not so much, Kenya is ahead of us Gov have removed new order mining rights info from website & water quality data
  • South Africa – info commissioner an important part to getting more transparency
  • South Africa – 1st hackathon in Aug 2012, secrecy bill attempting to shut down access to data, civil society active, OGP work too
  • Spain – lot of open data portals (~20), diverse, some good, some not. National portal is good but not much data. Big community tho
  • UK – 8661 datasets on new site, good stats, worked with openspending ppl to do reporting & better tools. Increase in public trust
  • Uruguay – Fascinating, gov working on data but dropping ball on other opengov #okfest
  • US – several initiatives out of date and not detailed enough
  • US – lots of stuff proposed, data laws, most of the work is overshadowed by Presidential election, OGP commitments being worked on.

Open Data Census – expand to capture information about individual initiatives?

OKFN have done a great job trying to get a useful comparative analysis of open data in countries around the world. I suggested it might be worthwhile to consider individual initiatives to get some understanding of exactly what is being done around the world, find commonalities and get some ideas around good practice in this space, especially as it is such a new area for so many people.

I put up some draft questions for the next Open Data Census and it’d be great to get feedback.

Outstanding talks I heard

There were many, many, outstanding talks at OKFestival. I’ll just briefly wrap up a few outstanding ones that I really enjoyed 🙂

Dr. Nagy-Rothengass from the European Commission

European Commission presentationDr Nagy-Rothengass gave a fascinating talk about the European Commission commitment to Open Data. They have committed substantial funding for this, around 100 million Euro and their core rationale for supporting open data are as follows:EU slide on case for open data

  1. Untapped business and economic opportunities: data is the new gold; possible direct and indirect gains of 140b Euro across the EU27
  2. Better Governance and citizen empowerment: open data increases transparency, citizen participation and administrative efficiency and accountability.
  3. Addressing societal challenges: data can enhance sustainability of health care systems, essential for tackling environmental challenges.
  4. Accelerating scientific progress: e-science essential for meeting the challenges of the 21st century in scientific discovery and learning.

Hans Rosling

Hans Rosling was a brilliant keynote.

My favourite quotes from the talk:

  • If you want to communicate with people you need to learn from tabloids. They are good at connecting with people.
  • The problem is not that people don’t know anything about the world, the problem is they have a completely incorrect view. evidence & statistics show world population growing to about 10b after which it normalises.
  • The western world has a toxic combination of arrogance and ignorance. Also gender equality doesn’t just happen. It requires work.
  • D3 d3js.org as a great tool for #datavis
  • We can’t rely on the leaders to deal with the money. We need to get involved and see for ourselves.
  • Life has never been so good as today. That the world is bad today doesn’t mean it hasn’t become enormously better.
  • We need to seriously invest in renewable energies & isn’t about polar bears. We are up against something much bigger.
  • You have to demand access to the data. Countries should report & need to release big data so we can do better.
  • OECD *sell* their data! We need to have it liberated so we can understand and learn.
  • Don’t talk about what you should do, just mock up and prototype.

He went a little through his normal developing countries vs developed countries spiel which clearly demonstrates the world is a lot closer statistically than many people believe. He spoke about world population growth and showed that it is one of very few things that statisticians have been consistently correct about (with only a 6ish% deviation from projections over 50 years), and yet there is still a lot of fear and misinformation about population growth. He said based on projections and the massive slowdown of population growth, that the world population would peak at around 10 billion and then that number would largely be sustained, unless there was an enormous disaster.

He showed the importance of not dividing the world up into “developing” and “developed” and that people’s understanding of the world is typically quite out of date, based on figures and perspectives taught in school but not updated throughout adult life. This leads to a community making decisions based on outdated information which leads to bad decisions. It was humbling to see actual statistics and realise that we don’t really have embedded societal mechanisms to update what is taught at school, and how this creates a perception of other countries and cultures that may fall completely out of date within just a few years.

Personally I believe strongly that it is through global collaboration that we can leapfrog issues and many of the attendees from what are traditionally call “developing” nations had great stories to tell of citizen empowerment and leapfrogging “developed” countries.

Hans core messages so far as I understood them included the importance of open data to make good decisions, the importance of recognisiing that our understanding of the world is usually out of date, and the importance of the active engagement of civil society in international and national matters to balance out the imbalance of power.

This last point he demonstrated very effectively by showing a picture of world leaders at the Summit on Financial Markets and the World Economy, with some information about which countries were loaning/giving money to others. It was fascinating. Aid money being given from one country to a “developing” country, which was in turn was loaning $30b to the US (when George Bush was in), who was in turn supporting them to get a seat on a UN council. It goes round and round.

Hans made a strong point that people should demand the data and transparency so they can make more informed decisions as a community and not just leave things up to world leaders.

Click through on the following thumbnails to see larger versions.

Hans Rosling slideHans Rosling slideHans Rosling slideHans Rosling slideHans Rosling slideHans Rosling slideHans Rosling slideHans Rosling slide

Rufus Pollock

Rufus is the Director and co-founder of OKFN, and quite an impressive figure. He is passionate about open data and is credited by Sir Tim Berners-Lee as being behind the “Raw Data Now” agenda. Rufus gave a great opening keynote where he spoke about the importance of open data combined with analysis and action. He said we have now started to see more and more data being opened up but if we don’t combine this with good analysis and then action in response to the analysis, then we will not see the benefits of open data.

His speech was largely a spoken version of his blog post called Managing Expectations II: Open Data, Technology and Government 2.0 – What Should We, And Should We Not Expect so I recommend you check it out 🙂

Diagram from Rufus Pollock on a theory of change
Diagram from Rufus Pollock on a theory of change

My contributions to OKFestival

Just a couple of notes for people I met there on my contributions.

I hosted a panel on open government, I contributed to several forums and I spoke on the closing panel with Philip Thigo (Program Associate at SODNET & Co-Founder of INFONET, Nairobi, Kenya) & Antti Poikola (OKF Finland Incubating Chapter, Helsinki) which gave me a good opportunity to further discuss the role of the public service in open government. This was received really well and I have a load of public service colleagues now from all around the world in this space.

On a panel about open data and culture at OKFest 2012 with Philip Thigo, Program Associate at SODNET & Co-Founder of INFONET, Nairobi, Kenya & Antti Poikola, OKF Finland Incubating Chapter, Helsinki. Photo from http://www.flickr.com/photos/tuija/8008433837/
On a panel about open data and culture at OKFest 2012 with Philip Thigo, Program Associate at SODNET & Co-Founder of INFONET, Nairobi, Kenya & Antti Poikola, OKF Finland Incubating Chapter, Helsinki. Photo from http://www.flickr.com/photos/tuija/8008433837/

I spoke a little bit about public engagement on public policy and mentioned the Public Sphere methodology that I developed with Senator Kate Lundy. The most recent consultation was one on Digital Culture which was a major contribution to the impending National Cultural Policy. A lot of people asked about analysis, so I spoke a little about the importance for both data analysis and “network” analysis of a consultation to ensure the outcomes have the appropriate context. See the Analysing the community of a public sphere blog post on Senator Lundy’s site.

I also mentioned my Distributed Democracy idea which a few people liked 🙂

Other links of possible interest:

GovHack and App4Country discussion

I was involved in a wonderful discussion with people from over 17 countries who do “apps” competitions and hackfests. It was great to hear about their initiatives and to share the lessons learnt from GovHack. Many of them expressed a lot of interest in our model which is a little broader than the “Apps4Country” model which has been quite popular in Europe. Most of them had the same problems with sustainability, longevity of outcomes from the hackfests, getting the government actively engaged. It was fascinating.

There are some good notes from the global hackfest/apps events collated here and there is a global mailing list (not very active atm) at http://lists.okfn.org/mailman/listinfo/appsforx.

The notes I made for my presentation about GovHack:

  • Narrative, design *and* hacks
  • Not focused on apps, but rather hacks (apps, mashups & datavis) – often applications emerge but “apps development” creates confusion with mobile vs web vs devel vs datavis
  • GovHack was part of a trilogy of events – GovCamp to set the narrative and vision, GovUX/Jam to apply design thinking to service delivery challenges in the public service, GovHack for open hacking and to make some service delivey design outcomes real
  • Open Government, Science & Digital Humanities – to add Data Journalism
    – Amazing how much of an impact it made, has really fired the imagination of the public and sector.
  • Enormous enthusiasm from the gov involved, 7 departments and agencies from federal and state governments were deeply engaged.
  • People flew from all over Australia to the two locations that we were simultaneously running the hackfest to participate.
  • Mentors from data owners and technologists to support teams along with sessions.
  • Made the documentation and presentation of the hack part of the judging criteria, which compelled teams to nicely capture content about their hacks which meant a good archive of the event.

Motivations:

  • Bring community together
  • Demonstrate value of open data
  • Raise the bar for the narrative in Aus, focus efforts on constructive efforts
  • Open the data, give depts buy in, connect their tech with community and leaders with success
  • Create new ways to do service delivery that can be integrated into gov, fundamentally disrupt gov expectations around “innovation”
  • Volunteer run which gave it extra credibility, buy in, and public engagement
  • A lot of bad expectations of “apps competitions” because of events that have done it badly, in Aus and internationally
  • Open Sourced hacks for people/companies/students/gov to build upon

Lessons learnt:

  • Hackers are motivated so long as you create some importance, and engage in conversation to manage tone and deal constructively with trolls
  • Prize money is helpful, but need to be careful to ensure good community, tone, “spirit of govhack” award
  • Scaling to go national – hackfest for two days, 3 days to vote, awards ceremony, followup 6 months later.
  • More funding would be useful
  • Ensure non tech elements encouraged, some great non “app” outcomes, eg the jewellery hack
  • Engage with the startup and VC sector, open sourcing outcomes means govhack can be yearly incubator for spin offs as well as input to gov. Startups love it as it is the best form of publicity
  • Non geeky hacks are the most reportable – History in ACTION
  • Technologists have a lot to contribute to policy, and there is a lot of work now to bring these groups together. Data visualisation and other uses of data can massively contribute to policy development and better policy outcomes.
  • Ongoing community engagement could be achieved through launching OKFN Au chapter to bring together communities across the gov, data journalists, hacker/geek communities and academia/research.

Interesting thoughts from “apps” conversation:

  • Need to strongly socialise – Finland
  • Apps for Europe, Spain, lessons from @aabella: 1) Civil society not politicians. Pollies have a role but it needs to be civil society driven. 2) Need to target general population not just tech community, get broader community involved.

Some additional links collected from the week of interest

Some random Open Science links sent:

Thanks

For the first time I tried couch surfing on this trip and stayed with a lovely Helsinki resident called Tarmo. A huge thanks to Tarmo for having me for the week, it was great to meet and I have to say, the couchsurfing culture is really friendly and lovely 🙂

Also, a huge thanks to Rufus for encouraging me to come, to Daniel Dietrich for his dedication to the open government space, and all the lovely people I met. I look forward to next year 🙂