quintel / atlas Goto Github PK

I'm currently probing around in Atlas to get a bit of a feeling where the scaling and duplication should take part.

Technically:

module Atlas
  class Scaler
    def initialize(area, dataset, number_of_residences)
      @area = area
      @dataset = dataset
      @number_of_residences = number_of_residences
    end
    
    def scale
      # write a new dataset 
      #   1. Duplicate all files that don't need direct scaling 
      #      (if possible and within scope of Atlas? - or should ETEngine handle the actual writing of the files?)
      #   2. Scale what needs to be scaled [1] and write the new data to those files

      new_dataset = Dataset.new(key: @area)
      new_dataset.path = "#{ @area } / #{ @area }"
      new_dataset.save
    end
  end
end

[1] = #55

The graph is as specified in a yaml format. But in which format must this graph be in order for it to be actually usuable for ETEngine.

I'm currently thinking that it's:

nodes:
  node_a:
    demand: 1
  node_b:
    demand: 1
edges:
  node_a-node_b@test:
    child_share: 1

But I'm not sure if this is the way to go? Also, I'm not sure which attribute to pick, is it always the case that it's demand, max_demand, child_share or parent_share? Or are there other attributes which need to be exported aswell?

There is a validation for the type already.

Once we have migrated away from the InputExcel completely, could we add this?

Currently, "sustainable" is boolean: true or false. But numerous carriers in the old dataset have values which are neither 0.0 nor 1.0:

• car_kms: nil
• cooling: nil
• coupling_carrier: nil
• diesel_mix: nil
• dumped_heat: nil
• electricity: nil
• gasoline_mix: nil
• gas_power_fuelmix: nil
• imported_steam_hot_water: 0.5
• light: nil
• loss: nil
• steam_hot_water: nil
• truck_kms: nil
• useable_heat: nil
• waste_mix: 0.38
• hot_water: nil

Carriers will need to be reimported.

Just so you know which .png to open first! Is it easy to implement this @antw?

Which attributes need to be scaled?

All attributes of a node:

:use: energetic
:energy_balance_group: technologies
:input:
  :crude_oil: 1.0
:output:
  :useable_heat: 0.788659793814433
  :loss: :elastic
:groups:
- :cost_traditional_heat
- :heat_production
:availability: 0.0
:free_co2_factor: 0.0
:forecasting_error: 0.0
:full_load_hours: 2190.0
:land_use_per_unit: 0.0
:takes_part_in_ets: 0.0
:part_load_efficiency_penalty: 0.0
:part_load_operating_point: 0.0
:electricity_output_capacity: 0.0
:heat_output_capacity: 5.0
:initial_investment: 300000.0
:ccs_investment: 0.0
:cost_of_installing: 0.0
:decommissioning_costs: 0.0
:fixed_operation_and_maintenance_costs_per_year: 5000.0
:variable_operation_and_maintenance_costs_per_full_load_hour: 0.0
:variable_operation_and_maintenance_costs_for_ccs_per_full_load_hour: 0.0
:construction_time: 0.0
:technical_lifetime: 15.0
:wacc: 0.1
:demand: 0.0

Attributes that need to be scaled:

:full_load_hours: 2190.0
:demand: 0.0

Attributes that might need to be scaled:

:part_load_efficiency_penalty: 0.0
:part_load_operating_point: 0.0
:wacc: 0.1

@ChaelKruip could you validate this?

When do we actually need to scale?

We currently persist the sparse-graph (i.e. pre-refinery) to a graph.yml file. The idea is now - when importing - to build the default graph (i.e. GraphBuilder.build) and merge the attributes from the sparse graph (from the graph.yml) into the attributes from the generated graph. This is already build this way.

Should we scale the graph attributes when importing? It only seems logical to do it this way.

1. Build graph
2. Overwrite values from graph.yml
3. Scale the attributes (from 1)
4. Done

There is a new in other projects to start using ActiveDocument, but without Atlas, also I think it would be a cool open source project, so we do something back to the Open Source community. 😉

Would like to hear your thoughts, @antw, @andremedeiros and @fvancamp.

When importing the graph, Refinery is stuck in an infinite loop when trying to calculate. It happens with these two slots:

[
  #<Refinery::Slot (in, electricity) node=:industry_steel_hisarna_consumption_useable_heat>,
  #<Refinery::Slot (in, useable_heat) node=:industry_steel_hisarna_consumption_useable_heat>
], 

0/1 # demand sum of all edges (which is zero)

Now industry_steel_hisarna_consumption_useable_heat has three 'carriers' (or slots?) not two. It also has a coupling carrier which @ChaelKruip and @jorisberkhout explained to me what they were. Now I also noticed that when exporting the graph.yml for that perticalar node (industry_steel_hisarna_consumption_useable_heat) has a cc_in attribute. When importing what should I do with that cc_in attribute? Is that even the cause of the infinite loop?

How do I properly debug this at all?

Currently, the scaling attribute is defined with

attribute :scaling,      Preset::Scaling

in DerivedDataset. But since the Preset::Scaling has attributes has_energy, has_agriculture and has_industry which are not respected when set for a DerivedDataset#scaling, this is not ideal - also, it is unnecessary, since the DerivedDataset has all the attributes of a Dataset directly, among them a lot of has_... attributes (has_energy is missing in the attributes of Dataset. Maybe we should add that? @antw @ChaelKruip).

Solution:
The Preset::Scaling class should get a newly invented superclass Atlas::Scaling without the has_.... This superclass is then used for the DerivedDataset#scaling.

There's currently a problem - as far as I understand it - with the interconnector_capacity attribute. The thing is that this attribute is used several times to determine the demand of certain nodes. The demand of the nodes however are exported to the graph.yml. So when you change the interconnector_capacity of a local dataset in the *.ad file, your changes will not be visible.

A solution I thought of would be to not have the interconnector_capacity inside of the *.ad file at all becuase it already implicitly lives inside of the graph.yml.

Even stronger, I'd raise a warning/error if somebody defines it inside of the local dataset since adding it won't do anything to your graph.

Cases:

1. Test scaling with rake scale (does it make sense, if not why etc.)
2. Create a scenario from ETModel
3. Change any number in the scaled ad file (test if it changes the numbers in ETModel).
4. Change an initializer input (test if it changes the numbers in ETModel)
   ...

Feedback:

Currently the local dataset "inherits" all the csv files from the national dataset. This should be the case? Or not be the case.

It should at least be possible to overwrite them. I think that this can work as:

datasets/ameland
datasets/nl
├── load_profiles
│   └── wind_offshore.csv

In this case the wind_offshore profile of nl is used.

datasets/ameland
├── load_profiles
│   └── wind_offshore.csv
datasets/nl
├── load_profiles
│   └── wind_offshore.csv

In this case the wind_offshore of ameland is used.

Here's what I came up with for dumping the graph YAML. Technically:

Runner.new(@dataset, GraphBuilder.new.graph).calculate

# And than transform this to a YAML file probably :) but first:

Still when I try to execute this I get an error based on the fixtures:

   Refinery::IncalculableGraphError:
       Insufficient data to calculate 1 items:

         * NodeDemand for #<Refinery::Node key=:my_residence>

How to resolve this?

I'm not sure what to scale in order to create a full correct local dataset. I now only scale the attributes of a dataset which have the proportional: true flag in the attributes. But I'm not sure that's all you need to scale.

This is assuming that you already scaled the exported demands, max_demands, etc. from the graph.yml.

Related discussion: quintel/etmodel#2324 (comment)

It would be great if Atlas were to know if all the filenames of inputs are unique.

The rake import task (original ticket: #30) chokes on the file .DS_Store.

When I run rake import on etsource or atlas, I get the following error message:

➜  atlas git:(master) rake import
cd ../etsource
rake aborted!
File exists - data/datasets/.DS_Store
/Users/quintel/Projects/etsource/Rakefile:38:in `block (2 levels) in <top (required)>'
/Users/quintel/Projects/etsource/Rakefile:28:in `each'
/Users/quintel/Projects/etsource/Rakefile:28:in `block in <top (required)>'
Tasks: TOP => import

I can circumvent this, by specifying a country: rake import DATASET=....

I noticed that ZeroDisabledSectors only disables nodes based on their groups but not based on their sector. Additionally, metal_industry is the only group that will ever get disabled.
Probably some more of the feature flags of Dataset should lead to disabling of nodes based on their sector!?

Should init. inputs be a separate class InitializerInput or should we continue using Input?
And if separate, then should InitializerInput and Input share a common base class? Maybe that would be called Input then and what is now Input becomes UserInput!?

There was some discussion about this in another issue already

As described here it should be possible to change the local dataset to the circumstances as they exist in that certain municipality/city/area. To do this we had the idea to adjust values using "initializer inputs" (the queries that live inside of, adjust_scaling/**/*.ad files).

My question:

Should it be possible to overwrite anything in the graph?

We export the "sparse graph" (i.e. the graph before refinery calcultates the rest of the demands). Should it therefor only be possible to alter parts of the sparse graph? The upsides to this approach is that refinery will calculate a valid graph in the end (so, there won't be any calculation errors). The downside is that the level of control you have over the graph is entirely determined by refinery and demands might trickle down in a way that's not desired [? if that's even a thing].

When you can alter anything in the graph (assuming therefor that the graph is first fully calculated in Refinery and than these 'edits' are overwriting the values in the graph) the downside is that Refinery has a chance of not calculating correctly (i.e. demands might not add up in combination with the shares etc.). This can be fixed by expanding the validate task in Atlas and make sure that everyting runs as planned when these initial inputs are set (so set initial input of x, run rake validate <OPTIONS/> and repeat). The upside is that you'd have more control over the graph. Than again you probably need to edit a lot of shares and nodes manually when changing the demand of a certain node in the middle of the fully calculated graph.

We're nearing the end of the stage0 phase of the local dataset project. The scaling is being reviewed as we speak and the basic implementation of the initializer inputs is in a pull request as well.

After both these pull requests are merged I think we need @DorinevanderVlies to test this locally on her machine to see if the workflow makes sense and if anything odd comes up.

Open todo (in somewhat prioritized order):

Refer to planning document

Running the new "bundle exec rake debug" task doesn't create/update the static.yml file. Is this a bug or a feature? How do I get etengine and model working again? @antw

Running

rake "debug:subgraph[../etsource/data,transport]" --trace

gives me the following output:

rake "debug:subgraph[../etsource/data,transport]" --trace
** Invoke debug:subgraph (first_time)
** Execute debug:subgraph
Setting up graph structure...
Setting up Refinery graph...
rake aborted!
wrong number of arguments (2 for 1)
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/bundler/gems/refinery-f1ca52a644ee/lib/refinery/diagram/base.rb:30:in draw_to' /Users/kruip/Projects/atlas/lib/tasks/debug.rb:110:inblock (2 levels) in <top (required)>'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:in call' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:inblock in execute'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:in each' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:inexecute'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:175:in block in invoke_with_call_chain' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:168:in invoke_with_call_chain' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:161:ininvoke'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:149:in invoke_task' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:inblock (2 levels) in top_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in each' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:inblock in top_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:115:in run_with_threads' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:100:intop_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:78:in block in run' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:165:instandard_exception_handling'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:75:in run' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/bin/rake:33:in<top (required)>'
/Users/kruip/.rbenv/versions/1.9.3-p194/bin/rake:23:in load' /Users/kruip/.rbenv/versions/1.9.3-p194/bin/rake:23:in

When we tried to delete all the factors from the inputs (since they are like a 'wax nose'), we discovered that Tome does not write back the comments.

-# changed max value for slider that was set to the output of the CHP:
-# future:Q(max_percentage_heat_network_agriculture)
-# This was not good, as tapped heat can also be produced centrally.
-# This central production is now also included in the named query.
-
-- factor = 100.0

Currently, there are defined in ConverterGroups sheet of the InputExcel. The xls2yaml script ports them to a yaml file that lives in etsource.

including @antw. Confer quintel/etengine#576.

From: #87 (comment)

How would you guys feel about naming these two new classes simply "Derived" and "Full"? "Dataset" already appears in the module tree, and it would shorten the .ad file names ("datasets/nl/nl.full.ad" instead of "datasets/nl/nl.full_dataset.ad").

• Rename classes in Atlas
• Rename files in ETSource

For the sake of consistency should the rake scale task also be taking FULL and DERIVED as arguments? Or should they still be called FULL_DATASET. I personally think that the _DATASET suffix provides some more context than just FULL.

In this case, the CENTRAL_PRODUCTION(self.key) could be used automatically, right @antw?

This ticket describe the functionality required to model the Industry for both The Netherlands and Germany.

The Problem

1) industry_transformation_generic_coal

Part of the coal use in the Industry is used for transformation purposes. This is not counted in the final demand of coal, but result in primary energy requirements of coal. When the metal industry is switched on, this is modeled almost correctly for NL. However, when the metal industry will be switched off, this coal requirement need to be accounted for in the industry_transformation_generic_coal converter by setting the output.coal conversion (coal to final demand) and the output.loss conversion (coal that to be modeled).

This efficiency is dependent on 1) whether or not the metal industry is specified, 2) the quality of the metal analysis.

See also https://github.com/quintel/rdr/issues/106 for the research background.

2) Blastfurnaces and cokesovens

The efficiencies of blastfurnaces and cokesovens are quite dependent on local circumstances. Therefore, e.g. input.coal, input.cokes and output.coal_gas are calculated in the analysis and currently set in the respective converters. However, it is most likely that the efficiencies for Germany will differ from the efficiencies of The Netherlands. Also efficiencies between years can differ.

See also https://github.com/quintel/rdr/issues/107 for the research background.

Potential solution

A potential solution for the technical part of this problem (as @antw suggest in https://github.com/quintel/rdr/issues/64#issuecomment-21788427) is to query the efficiencies based on a table that is generated in a analysis that is output from a analysis, e.g. in case of industry_transformation_generic_coal for NL 2011:

~ output.coal = EFF(…, output.coal)
~ output.loss = EFF(…, output.loss)

@ChaelKruip showed me the initial idea for creating local datasets (which was a picture on a whiteboard - in a circular movement) which was drawn by @antw

I remembered him telling me explicitly that a local dataset needs to be put in a single .ad-file.
But I was wondering what the reasoning behind that is?

Wouldn't it be easier to just keep the data structure exactly the same to a non-local dataset?

see quintel/etsource#328

Currently, there are still some problems with initializing the whole graph with solely final demands from the energy balance and related parent and child shares. To solve https://github.com/quintel/rdr/issues/105 we need to query a demand from a csv file.

A proposed syntax is ~ demand = DEMAND(filename, node_key). The values in the csv files will be in TJ. It would be nice if we can store the demand files in for example etsource/data/datasets/country/demands/....

The vast majority of slots in the graph have only a single attribute: the conversion (called share in Atlas/Refinery). However, a small number have an additional attribute: reset_to_zero.

The Node documents provide no way for us to add additional attributes for a slot; you can only specify the conversion:

~ input.gas = 0.5

I'm unsure how to go about supporting extra attributes for slots. I have an idea, but it may not be necessary as the only extra attribute I can find is reset_to_zero, and it appears to be used only on the four input slots for "coupling_carrier".

@dennisschoenmakers @ChaelKruip Instead of trying to shoehorn slot attributes into Atlas, do you think it would be reasonable to just have the ETEngine importer assign reset_to_zero: true to all "coupling_carrier" input slots?

@ChaelKruip proposed an alternative idea for the initializer inputs and I want to test how feasible this is.

The current idea is to make initializer inputs behave like user_values. So:

[ Atlas::GraphDeserializer / Atlas::Runner]
          |
          V
[ Atlas::FullExporter ]
          |
          V
[ Load data from dumped MessagePack ]
          |
       [ ... ]
          |
          V
[ Overwrite inputs with initializer inputs ] [?]

As an expansion/extra-step to that idea @ChaelKruip proposed the following:

                  O B
                 / 
                / (A-B share: 0.5)
A (demand = 5) O 
                \ (A-C share: 0.5)
                 \ 
                  O C

Than this is the full-graph

                  O B (demand = 2.5)
                 / 
                / (A-B share: 0.5)
A (demand = 5) O 
                \ (A-C share: 0.5)
                 \ 
                  O C (demand = 2.5)

If a researcher than knows the demand of node/converter B and sets it to for example (10). This would work the same as the idea proposed above (as user_values).

                  O B (demand = 10)
                 / 
                / (A-B share: 0.5)
A (demand = 5) O 
                \ (A-C share: 0.5)
                 \ 
                  O C (demand = 2.5)

The extra step
Than the 'altered' sparse graph (if assuming the sparse graph will have the same starting points and edges) will look like:

                  O B
                 / 
                / (A-B share: 0.5)
A (demand = 20) O 
                \ (A-C share: 0.5)
                 \ 
                  O C

Should this be reflected in the graph.yml? Is that useful?

It would be useful - before running everything through ETEngine - to be able to check through git which numbers will change.

Refinery completely ignores coupling_carrier when performing it's calculations – a requirement for outputting a valid graph. Yet, ETEngine still requires that we provide a conversion for those slots.

As a temporary solution, I added the old InputExcel coupling carrier conversions, however these numbers are so specific I suspect they might be the result of some calculation in InputExcel rather than a hand-edited number.

• If they are hand-edited, and not region-specific, then the numbers can be left in the .ad file.
• If they are the result of a calculation, but require only data already available in CSVs, then we can define them as a query.
• If they are the result of a calculation, but CSVs aren't sufficient (the conversions depend on demands or shares from Refinery) then we'll have to define them as queries, and (somehow) special-case them to run after Refinery.

@ChaelKruip Could you shed some light on how you'd like to handle these slots?
@dennisschoenmakers Please jump in if you have any opinions on this.

When I am running Atlas on the PL 2012 dataset (quintel/etsource@new_countries) I get the following validation error:

➜  atlas git:(master) rake debug DATASET=pl

Post-calculation validations failed with the following errors:

  * #<Refinery::Slot (out, waste_mix) node=:energy_distribution_biogenic_waste> has an undetermined share
  * #<Refinery::Slot (out, biogenic_waste) node=:energy_distribution_biogenic_waste> has an undetermined share

This is most likely cause by the zero share for biogenic waste in the energy_distribution_waste_mix_child_share file.

@antw Any idea to solve this validation error?

Output of tree etsource/datasets/example

datasets/example
├── central_producers.csv
├── demands
│   └── metal_demands.csv
├── efficiencies
│   ├── energy_cokesoven_consumption_coal_gas_efficiency.csv
│   ├── energy_cokesoven_transformation_coal_efficiency.csv
│   ├── energy_mixer_for_gas_power_fuel_efficiency.csv
│   ├── energy_steel_blastfurnace_bat_transformation_cokes_efficiency.csv
│   ├── energy_steel_blastfurnace_current_transformation_cokes_efficiency.csv
│   ├── industry_chemicals_fertilizers_steam_methane_reformer_hydrogen_efficiency.csv
│   ├── industry_refinery_transformation_crude_oil_efficiency.csv
│   ├── industry_steel_blastfurnace_bat_consumption_useable_heat_efficiency.csv
│   ├── industry_steel_blastfurnace_current_consumption_useable_heat_efficiency.csv
│   └── industry_transformation_generic_coal_efficiency.csv
├── energy_balance.csv
├── energy_balance.gpg
├── example.ad
├── load_profiles
│   └── total_demand.csv
├── primary_production.csv
├── shares
│   ├── agriculture_final_demand_electricity_parent_share.csv
│   ├── buildings_final_demand_coal_parent_share.csv
│   ├── buildings_final_demand_crude_oil_parent_share.csv
│   ├── buildings_final_demand_electricity_parent_share.csv
│   ├── buildings_final_demand_for_cooling_electricity_parent_share.csv
│   ├── buildings_final_demand_for_lighting_electricity_parent_share.csv
│   ├── buildings_final_demand_for_space_heating_electricity_parent_share.csv
│   ├── buildings_final_demand_for_space_heating_network_gas_parent_share.csv
│   ├── buildings_final_demand_network_gas_parent_share.csv
│   ├── buildings_final_demand_wood_pellets_parent_share.csv
│   ├── buildings_local_production_electricity_parent_share.csv
│   ├── buildings_useful_demand_after_motion_detection_light_child_share.csv
│   ├── buildings_useful_demand_cooling_child_share.csv
│   ├── buildings_useful_demand_for_space_heating_child_share.csv
│   ├── buildings_useful_demand_light_child_share.csv
│   ├── energy_distribution_hydrogen_child_share.csv
│   ├── energy_distribution_waste_mix_child_share.csv
│   ├── energy_distribution_wood_parent_share.csv
│   ├── energy_locally_available_hydrogen_before_p2g_child_share.csv
│   ├── energy_national_gas_network_natural_gas_child_share.csv
│   ├── energy_national_gas_network_natural_gas_greengas_child_share.csv
│   ├── industry_aluminium_burner_network_gas_parent_share.csv
│   ├── industry_final_demand_coal_non_energetic_parent_share.csv
│   ├── industry_final_demand_coal_parent_share.csv
│   ├── industry_final_demand_crude_oil_non_energetic_parent_share.csv
│   ├── industry_final_demand_crude_oil_parent_share.csv
│   ├── industry_final_demand_electricity_parent_share.csv
│   ├── industry_final_demand_for_chemical_coal_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_chemical_coal_parent_share.csv
│   ├── industry_final_demand_for_chemical_crude_oil_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_chemical_crude_oil_parent_share.csv
│   ├── industry_final_demand_for_chemical_electricity_parent_share.csv
│   ├── industry_final_demand_for_chemical_network_gas_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_chemical_network_gas_parent_share.csv
│   ├── industry_final_demand_for_chemical_steam_hot_water_parent_share.csv
│   ├── industry_final_demand_for_chemical_wood_pellets_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_chemical_wood_pellets_parent_share.csv
│   ├── industry_final_demand_for_metal_coal_parent_share.csv
│   ├── industry_final_demand_for_metal_electricity_parent_share.csv
│   ├── industry_final_demand_for_metal_network_gas_parent_share.csv
│   ├── industry_final_demand_for_metal_steam_hot_water_parent_share.csv
│   ├── industry_final_demand_for_other_coal_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_other_coal_parent_share.csv
│   ├── industry_final_demand_for_other_crude_oil_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_other_crude_oil_parent_share.csv
│   ├── industry_final_demand_for_other_electricity_parent_share.csv
│   ├── industry_final_demand_for_other_food_electricity_parent_share.csv
│   ├── industry_final_demand_for_other_network_gas_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_other_network_gas_parent_share.csv
│   ├── industry_final_demand_for_other_paper_electricity_parent_share.csv
│   ├── industry_final_demand_for_other_steam_hot_water_parent_share.csv
│   ├── industry_final_demand_for_other_wood_pellets_non_energetic_parent_share.csv
│   ├── industry_final_demand_for_other_wood_pellets_parent_share.csv
│   ├── industry_final_demand_network_gas_non_energetic_parent_share.csv
│   ├── industry_final_demand_network_gas_parent_share.csv
│   ├── industry_final_demand_steam_hot_water_parent_share.csv
│   ├── industry_final_demand_wood_pellets_non_energetic_parent_share.csv
│   ├── industry_final_demand_wood_pellets_parent_share.csv
│   ├── industry_steel_blastfurnace_burner_coal_gas_parent_share.csv
│   ├── industry_transformation_generic_coal_parent_share.csv
│   ├── residences_final_demand_coal_parent_share.csv
│   ├── residences_final_demand_crude_oil_parent_share.csv
│   ├── residences_final_demand_electricity_parent_share.csv
│   ├── residences_final_demand_for_appliances_electricity_parent_share.csv
│   ├── residences_final_demand_for_cooking_electricity_parent_share.csv
│   ├── residences_final_demand_for_cooling_electricity_parent_share.csv
│   ├── residences_final_demand_for_hot_water_electricity_parent_share.csv
│   ├── residences_final_demand_for_hot_water_network_gas_parent_share.csv
│   ├── residences_final_demand_for_lighting_electricity_parent_share.csv
│   ├── residences_final_demand_for_space_heating_electricity_parent_share.csv
│   ├── residences_final_demand_for_space_heating_network_gas_parent_share.csv
│   ├── residences_final_demand_network_gas_parent_share.csv
│   ├── residences_final_demand_steam_hot_water_parent_share.csv
│   ├── residences_final_demand_wood_pellets_parent_share.csv
│   ├── residences_new_houses_useful_demand_for_cooling_child_share.csv
│   ├── residences_new_houses_useful_demand_for_heating_child_share.csv
│   ├── residences_old_houses_useful_demand_for_cooling_child_share.csv
│   ├── residences_old_houses_useful_demand_for_heating_child_share.csv
│   ├── residences_space_heater_micro_chp_network_gas_child_share.csv
│   ├── residences_useful_demand_for_cooling_after_insulation_parent_share.csv
│   ├── residences_useful_demand_for_hot_water_after_solar_heater_and_p2h_parent_share.csv
│   ├── residences_useful_demand_for_hot_water_for_houses_with_p2h_child_share.csv
│   ├── residences_useful_demand_for_hot_water_for_houses_with_solar_heater_child_share.csv
│   ├── residences_useful_demand_for_space_heating_after_insulation_for_houses_with_solar_heater_child_share.csv
│   ├── residences_useful_demand_for_space_heating_after_insulation_parent_share.csv
│   ├── residences_water_heater_fuel_cell_chp_network_gas_child_share.csv
│   ├── residences_water_heater_micro_chp_network_gas_child_share.csv
│   ├── residences_water_heater_solar_thermal_parent_share.csv
│   ├── transport_final_demand_bio_ethanol_parent_share.csv
│   ├── transport_final_demand_biodiesel_parent_share.csv
│   ├── transport_final_demand_diesel_parent_share.csv
│   ├── transport_final_demand_electricity_parent_share.csv
│   ├── transport_final_demand_for_road_electricity_parent_share.csv
│   ├── transport_final_demand_for_road_greengas_parent_share.csv
│   ├── transport_final_demand_for_road_natural_gas_parent_share.csv
│   ├── transport_final_demand_gasoline_parent_share.csv
│   ├── transport_road_mixer_compressed_network_gas_parent_share.csv
│   ├── transport_road_mixer_diesel_mix_parent_share.csv
│   ├── transport_road_mixer_gasoline_mix_parent_share.csv
│   ├── transport_road_mixer_lng_child_share.csv
│   ├── transport_shipping_mixer_diesel_child_share.csv
│   └── transport_shipping_mixer_lng_child_share.csv
└── time_curves
    ├── energy_distribution_woody_biomass_time_curve.csv
    ├── energy_extraction_coal_time_curve.csv
    ├── energy_extraction_crude_oil_time_curve.csv
    ├── energy_extraction_lignite_time_curve.csv
    ├── energy_extraction_natural_gas_time_curve.csv
    └── energy_extraction_uranium_oxide_time_curve.csv

5 directories, 125 files

What exactly needs to be scaled and what doesn't need to be scaled? I heared that @jorisberkhout recalled that there was an issue about this explaining that.

Files that needs to be changed when automatically generating a dataset:

etsource/datasets.yml (add a row with which dataset there is)
mv etsource/datasets/<local_dastaset>/<source_dataset>.ad -> etsource/datasets/<local_dastaset>/<local_dataset>.ad

In an attempt to resolves the issues https://github.com/quintel/etdataset/issues/607 and quintel/etsource#921 (comment), I introduced and renamed several shares and updated the edges accordingly.

A little background

After we merged this pull request the P2H node plays a very similar role in fulfilling the hot water demand as the add-on heat pump does in fulfilling the space heating demand. I therefore tried to replicate the add-on heat pump situation for the P2H node. Below is a screenshot of the graph (on BETA) and the initial shares that are defined for this part of the graph including their values (space heating in RED, hot water in BLUE and the shared solar water heater in green).

The issue

The space heating part of the graph (RED) shown below contains enough information to initialise the graph (as it always did). Interestingly, the hot water part (BLUE) contains the same amount of information, but results in incalculable parts of the graph when I run rake debug on Atlas. My guess was that this has to do with the righthand share of the blue part being 1-0. I changed this to 0.999-0.001 and this indeed resulted in a fully initialised graph with correct results.

The question

@antw, do you know why the 1-0 share results in an incalculable graph and how we can best resolve this? I would like to resolve this before the September deploy.

NB

I have only updated the shares for the NL dataset as doing it for all is quite tedious work. Any failing specs associated with other datasets are thus to be expected and I will resolve those once this issue is resolved.

Including @ChaelKruip .

After running all Research Analyses, a bunch of CSV files are created on etdataset that need to be copied to etsource. It would be great if this could be done automagically....

I made a list of the old and new file names (including directories) in an Excel workbook.

Please be aware of the following:

• it is good practice, to first delete the all files from etsource, before copying the fresh files
  • remove the energy balance from etsource
  • remove everything in the etsource/data/datasets/<country> folder, _except_ for the country.ad file
• The file directories on rdr change with every country and year
• The file directories on etsource change with every country (and possibly with every year if we ever want to enable that).
• The rake task should be easy to maintain. Future changes to the graph will make it necessary to export more/less CSV files from rdr to etsource.

Specify:

• rake task with number_of_residences as an argument, and a country like nl.
• automatically generate a scaled down dataset

The current way of "validating" a national dataset 'graph' is through rake debug. However, I can imagine that you'd want something similar for a local dataset.

See quintel/etsource#328

So you can do

Atlas::Dataset.create(properties)

Which calls new and save! Similar to ActiveRecord.

Atlas throws a really undescriptive error on this line, when somebody formats his or her query wrongly (cough, @RobTerwel). For instance

- query = EACH(

throws an error.

- query =
   EACH(

Doesn't.

It would be practical if a little more descriptive error is thrown stating that you can't do this and in which file this error occurs.

Now it's sometimes a bit hard to read when all different kinds of converters are standing listed on top of each other, like import, production, export, conversions, etc.

I'm not exactly sure how to do this in an elegant way though...

After importing the graph I compared beta and local by performing the following code in the sandbox:

Hash[@scope.graph.converters.map{|c| [c.key, c.demand] }]

+--------------------------------------------------------+--------------------+--------------------+---------------------+
| key                                                    | beta               | local              | difference          |
+--------------------------------------------------------+--------------------+--------------------+---------------------+
| energy_steel_blastfurnace_current_transformation_cokes | 89066589999.99962  | 0.0                | 89066589999.99962   |
| energy_distribution_cokes                              | 53608499999.99974  | 0.0                | 53608499999.99974   |
| energy_cokesoven_transformation_coal                   | 72821157188.91655  | 0.0                | 72821157188.91655   |
| energy_cokesoven_consumption_coal_gas                  | 7400469944.584353  | 0.0                | 7400469944.584353   |
| industry_locally_available_electricity                 | 141496247385.3905  | 141199400000.00006 | 296847385.3904419   |
| energy_distribution_coal_gas                           | 50610397143.576675 | 18299614029.199936 | 32310783114.37674   |
| energy_export_coal_gas                                 | 25207160555.182816 | 0.0                | 25207160555.182816  |
| energy_import_coal_gas                                 | 0.0                | 18299614029.199936 | -18299614029.199936 |
| energy_distribution_coal                               | 362651179619.91205 | 254371932430.99564 | 108279247188.91641  |
| energy_import_coal                                     | 362651179619.91205 | 254371932430.99564 | 108279247188.91641  |
| energy_power_mv_distribution_network_electricity       | 339955443308.01074 | 339658595922.6203  | 296847385.3904419   |
| energy_power_mv_transport_network_electricity          | 328822721390.2025  | 328525874004.8121  | 296847385.3904419   |
| energy_national_gas_network_natural_gas                | 1402481155592.2878 | 1402481155592.288  | -0.000244140625     |
| energy_power_transformer_mv_hv_electricity             | 328822721390.2025  | 328525874004.8121  | 296847385.3904419   |
| energy_power_hv_network_electricity                    | 359615461390.20245 | 359304650492.1222  | 310810898.08026123  |
| energy_interconnector_imported_electricity             | 65761861390.20233  | 65451050492.12213  | 310810898.0802002   |
| energy_import_electricity                              | 65761861390.20233  | 65451050492.12213  | 310810898.0802002   |
| energy_power_hv_network_shortage                       | 6.103515625e-05    | 0.0                | 6.103515625e-05     |
| energy_power_hv_network_loss                           | 16156109999.999962 | 16142146487.310165 | 13963512.689796448  |
| energy_export_network_gas                              | 1171278445596.1736 | 1171278445596.1733 | 0.000244140625      |
+--------------------------------------------------------+--------------------+--------------------+---------------------+

Some demands don't add up. I did filter out most rounding errors.

As specified here.