10 Nov Data Acquisition & Transformation Solution Enables Non-Profit to Provide Timely Food Security Services to SE Asian Government
As a data services consultancy, TenPoint7 helps organizations access their data assets and relevant external sources providing rich insights that then enable them to make impactful business decisions. We recently worked on such an opportunity serving a world-renowned non-profit organization that helped them further their noble mission.
About World Food Program
This non-profit was The World Food Program. WFP, a United Nations agency, is the world’s largest humanitarian agency broadly chartered with the mission to fight hunger worldwide. One approach that WFP heavily utilizes to fulfill this mission is through the active monitoring and pro-active resolution of food security crises around the world.
WFP’s ability to provide such food security assistance to governments is highly dependent on its on-going access to reliable and current food price data. It currently monitors the price of key food staples in about 1,200 markets across 75 countries through its Vulnerability Analysis and Mapping (VAM) unit.
To provide this type of assistance effectively, WFP uses a number of tools that include food price monitoring. To facilitate such analysis, the VAM unit launched a global food price data store in 2011, which centralizes data from all its field offices. This global price data store not only allows its internal (and external) researchers to conduct time-series analysis across all monitored countries, but it also includes an automatic early warning system called ALPS (Alert for Price Spikes) that monitors the extent to which a local food commodity market experiences unusually high food price levels.
By keeping a watchful eye on prices of staple foods across markets on a frequent basis, WFP is therefore able to highlight areas of emerging concern for timely governmental intervention.
WFP’s challenges in monitoring food prices in Indonesia
One such country for which WFP conducts price monitoring and food security is Indonesia.
WFP’s VAM unit in Indonesia currently partners with multiple federal government agencies that collect food prices at different time intervals and level of details including: the Ministry of Trade, Bureau of Statistics (BPS) and Food Security Agency (BKP). While VAM is able to collect and store price information from some of these agencies, it is not able to efficiently extract data from agencies such as BKP. Such data acquisition challenges therefore were limiting WFP’s ability to effectively monitor food security across Indonesia.
Specifically in the case of BKP, food price data from this source was highly valuable to WFP. This was primarily because BKP collects and publishes price data at a much more granular detail than other agencies in the country. Furthermore, both retail and wholesale food price data on 14 essential commodities across all 34 provinces and 273 districts of Indonesia is collected and made available on BKP’s website, a significantly richer price dataset than that provided by other government sources.
However, accessing and collecting BKP’s valuable price data was a highly cumbersome, manual and error prone task that would have consumed substantial WFP manpower. The current web interface to access BKP data was a severe limiting factor. In addition, basic backend server infrastructure also severely limited the performance of retrieving data from BKP via manual steps.
WFP Indonesia was therefore seeking a solution that would allow for the timely and efficient extraction of BKP food price data that would then be subsequently made available to its VAM price data store. If this could be achieved, the Government of Indonesia and WFP would then incrementally benefit from ALPS (and other WFP systems and tools) allowing them to respond to emerging food security crises in a timely manner.
Open source solution enabled WFP to collect & transform critical price data
Working closely with WFP Indonesia, we designed and implemented a solution based entirely on open source that achieved the desired outcomes of this project. The solution, implemented purely in Python (2.7) and utilizing common
Python libraries such as beautifulsoup and others, provided a simple and elegant Windows-based interface for WFP to efficiently extract weekly price data of all available commodities and across all Indonesian provinces from BKP’s site.
In addition, simple transformations of the collected raw price data was also conducted resulting in auto-generated Excel reports that contained relevant price data of all commodities. These reports subsequently provided a simple interface to allow price data to be added to the VAM data store.
The solution also automated extraction of historical price data stretching back to 3 years thereby enabling VAM researchers to conduct better time series and other analyses on food price data in Indonesia.
Timely and efficient data acquisition is the first step in the journey of generating insights that help organizations make better decisions. Whether acquisition needs to occur from internal data assets or external sites, it begins with the construction of a well-designed data pipeline, big or small, facilitating critical business analysis downstream. TenPoint7 was grateful to team with the World Food Program and help them continue their mission of providing food security in developing countries such as Indonesia.