Proposal for a script to merge regional meta and to manage CRUD functions for future components meta. Removes the need for any hand-editing of master meta file.

Meta merge & script proposal.md

Requirements & Features

• provide a metadata template which meta producers must follow
• easy to update: update entire components or specific properties of a component
• There is a set of generic meta for each region. New meta should ONLY include unique keys/properties. i.e.
• Radio has generic key of name, and unique keys of radio URL & sid. So meta producer would only provide URL & sid.
• namespace properties of a particular component, i.e.: weatherLocations, radioSid, radioUrl, sportTeams, travelSearchTerm
• Region URL is the key for each node
• script WILL NOT modify metadata provided. Must be in correct format.
• script can be provided either json or yaml, outputs TBC

Script functionality

• view meta for a particular region, i.e. rake meta:view '/news/england/london'
• add meta for a new feature/component and basic data formatting validation. Exit if not valid throwing a custom error
• add meta properties for an existing component
• warns when existing properties might be overwritten. Lets user continue or exit.
• delete all properties for a component, e.g. weather
• delete specific properties for a component (passed in as an array), e.g. weatherLocations
• delete generic properties e.g. name
• update existing property name, e.g. radioSid --> radioId

Examples

Properties are nested by component type e.g. radio

 "/news/england/beds_bucks_and_herts": {
     "radio": {
         "name": "Beds, Herts & Bucks",
         "radioSid": "bedshertsandbucks"
     }
 },

Properties are namespaced by component type (unless they are generic properties e.g. "name") and no nesting

 "/news/england/beds_bucks_and_herts": {
     "name": "Beds, Herts & Bucks",
     "radioSid": "threecounties",
     "travelSearchTerm": "bedshertsandbucks",
     "weatherLocations": {}
 }

Also, not sure there's need to mandate another file or file-snippet as the argument. Fact that we currently have multiple files now, doesn't mean we must put arguments into a file in a certain format in order to perform operations on the master file in the future.

@kkajero Agreed that we'll first focus on the task of merging all of the meta files together. And I also agree about adding in features as and when in the future. It could be a lot of effort to add all of the CRUD features when they may never be used, or rarely.

• yaml will be the output format
• we can use Sam's original Ruby script as a starting point
• we'll write some Ruby tests. Will find out what's best for a data manipulation script.

In answer to your question "How do we know when this becomes a bad solution?": tough to quantify this.

From a performance PoV: It should be highly cached with APC. Had a chat with John and he suggested using a rudimentary tool such as ab on sandbox, first on the merged file and then on a potentially much larger file (just by faking it with repeating the same data).

As a later optimisation, we could use something like Rake or Grunt to split out the meta based on region index. That way we can serve up e.g. england_derbyshire.yml (gzipped) which should make the payload a lot smaller than a massive merged file