quintel / atlas Goto Github PK
View Code? Open in Web Editor NEWRuby library for working with ETSource data
License: MIT License
Ruby library for working with ETSource data
License: MIT License
I'm currently probing around in Atlas to get a bit of a feeling where the scaling and duplication should take part.
Technically:
module Atlas
class Scaler
def initialize(area, dataset, number_of_residences)
@area = area
@dataset = dataset
@number_of_residences = number_of_residences
end
def scale
# write a new dataset
# 1. Duplicate all files that don't need direct scaling
# (if possible and within scope of Atlas? - or should ETEngine handle the actual writing of the files?)
# 2. Scale what needs to be scaled [1] and write the new data to those files
new_dataset = Dataset.new(key: @area)
new_dataset.path = "#{ @area } / #{ @area }"
new_dataset.save
end
end
end
[1] = #55
The graph is as specified in a yaml format. But in which format must this graph be in order for it to be actually usuable for ETEngine
.
I'm currently thinking that it's:
nodes:
node_a:
demand: 1
node_b:
demand: 1
edges:
node_a-node_b@test:
child_share: 1
But I'm not sure if this is the way to go? Also, I'm not sure which attribute to pick, is it always the case that it's demand
, max_demand
, child_share
or parent_share
? Or are there other attributes which need to be exported aswell?
There is a validation for the type
already.
Once we have migrated away from the InputExcel completely, could we add this?
Currently, "sustainable" is boolean: true or false. But numerous carriers in the old dataset have values which are neither 0.0 nor 1.0:
Carriers will need to be reimported.
Just so you know which .png to open first! Is it easy to implement this @antw?
All attributes of a node:
:use: energetic
:energy_balance_group: technologies
:input:
:crude_oil: 1.0
:output:
:useable_heat: 0.788659793814433
:loss: :elastic
:groups:
- :cost_traditional_heat
- :heat_production
:availability: 0.0
:free_co2_factor: 0.0
:forecasting_error: 0.0
:full_load_hours: 2190.0
:land_use_per_unit: 0.0
:takes_part_in_ets: 0.0
:part_load_efficiency_penalty: 0.0
:part_load_operating_point: 0.0
:electricity_output_capacity: 0.0
:heat_output_capacity: 5.0
:initial_investment: 300000.0
:ccs_investment: 0.0
:cost_of_installing: 0.0
:decommissioning_costs: 0.0
:fixed_operation_and_maintenance_costs_per_year: 5000.0
:variable_operation_and_maintenance_costs_per_full_load_hour: 0.0
:variable_operation_and_maintenance_costs_for_ccs_per_full_load_hour: 0.0
:construction_time: 0.0
:technical_lifetime: 15.0
:wacc: 0.1
:demand: 0.0
Attributes that need to be scaled:
:full_load_hours: 2190.0
:demand: 0.0
Attributes that might need to be scaled:
:part_load_efficiency_penalty: 0.0
:part_load_operating_point: 0.0
:wacc: 0.1
@ChaelKruip could you validate this?
We currently persist the sparse-graph (i.e. pre-refinery) to a graph.yml
file. The idea is now - when importing - to build the default graph (i.e. GraphBuilder.build
) and merge the attributes from the sparse graph (from the graph.yml
) into the attributes from the generated graph. This is already build this way.
Should we scale the graph attributes when importing? It only seems logical to do it this way.
graph.yml
There is a new in other projects to start using ActiveDocument, but without Atlas, also I think it would be a cool open source project, so we do something back to the Open Source community. ๐
Would like to hear your thoughts, @antw, @andremedeiros and @fvancamp.
When importing the graph, Refinery is stuck in an infinite loop when trying to calculate. It happens with these two slots:
[
#<Refinery::Slot (in, electricity) node=:industry_steel_hisarna_consumption_useable_heat>,
#<Refinery::Slot (in, useable_heat) node=:industry_steel_hisarna_consumption_useable_heat>
],
0/1 # demand sum of all edges (which is zero)
Now industry_steel_hisarna_consumption_useable_heat
has three 'carriers' (or slots?) not two. It also has a coupling carrier which @ChaelKruip and @jorisberkhout explained to me what they were. Now I also noticed that when exporting the graph.yml
for that perticalar node (industry_steel_hisarna_consumption_useable_heat
) has a cc_in
attribute. When importing what should I do with that cc_in
attribute? Is that even the cause of the infinite loop?
How do I properly debug this at all?
Currently, the scaling attribute is defined with
attribute :scaling, Preset::Scaling
in DerivedDataset
. But since the Preset::Scaling has attributes has_energy
, has_agriculture
and has_industry
which are not respected when set for a DerivedDataset#scaling, this is not ideal - also, it is unnecessary, since the DerivedDataset has all the attributes of a Dataset directly, among them a lot of has_...
attributes (has_energy
is missing in the attributes of Dataset
. Maybe we should add that? @antw @ChaelKruip).
Solution:
The Preset::Scaling
class should get a newly invented superclass Atlas::Scaling
without the has_...
. This superclass is then used for the DerivedDataset#scaling.
There's currently a problem - as far as I understand it - with the interconnector_capacity
attribute. The thing is that this attribute is used several times to determine the demand of certain nodes. The demand of the nodes however are exported to the graph.yml
. So when you change the interconnector_capacity
of a local dataset in the *.ad
file, your changes will not be visible.
A solution I thought of would be to not have the interconnector_capacity
inside of the *.ad
file at all becuase it already implicitly lives inside of the graph.yml
.
Even stronger, I'd raise a warning/error if somebody defines it inside of the local dataset since adding it won't do anything to your graph.
Cases:
rake scale
(does it make sense, if not why etc.)Feedback:
Currently the local dataset "inherits" all the csv files from the national dataset. This should be the case? Or not be the case.
It should at least be possible to overwrite them. I think that this can work as:
datasets/ameland
datasets/nl
โโโ load_profiles
โย ย โโโ wind_offshore.csv
In this case the wind_offshore
profile of nl
is used.
datasets/ameland
โโโ load_profiles
โ โโโ wind_offshore.csv
datasets/nl
โโโ load_profiles
โ โโโ wind_offshore.csv
In this case the wind_offshore
of ameland
is used.
Wouters-MacBook-Air:atlas wmeyers$ bundle exec rake debug:graph
Creating "initial values" diagrams:
rake aborted!
Error from "/usr/local/bin/dot" -q1 -Tpng -otmp/debug/agriculture.0-initial-values.png /var/folders/zd/vss6_qjx39ncl7x_ct51vsz40000gn/T/graphviz.rb20130814-3722-1a0pswx:
dyld: Library not loaded: /usr/lib/libltdl.7.dylib
Referenced from: /usr/local/bin/dot
Reason: image not found
/Users/wmeyers/.rvm/gems/ruby-1.9.3-p392/gems/ruby-graphviz-1.0.9/lib/graphviz/utils.rb:77:in `output_from_command'
/Users/wmeyers/.rvm/gems/ruby-1.9.3-p392/gems/ruby-graphviz-1.0.9/lib/graphviz.rb:589:in `output'
/Users/wmeyers/.rvm/gems/ruby-1.9.3-p392/bundler/gems/refinery-611adcb28cc1/lib/refinery/diagram/base.rb:46:in `draw_to'
/Users/wmeyers/development/atlas/lib/tasks/debug.rb:14:in `block (2 levels) in draw_diagrams'
/Users/wmeyers/development/atlas/lib/tasks/debug.rb:13:in `each'
/Users/wmeyers/development/atlas/lib/tasks/debug.rb:13:in `block in draw_diagrams'
/Users/wmeyers/development/atlas/lib/atlas/term/reporter.rb:40:in `report'
/Users/wmeyers/development/atlas/lib/atlas/term/reporter.rb:17:in `report'
/Users/wmeyers/development/atlas/lib/tasks/debug.rb:11:in `draw_diagrams'
/Users/wmeyers/development/atlas/lib/tasks/debug.rb:60:in `block (2 levels) in <top (required)>'
/Users/wmeyers/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `eval'
/Users/wmeyers/.rvm/gems/ruby-1.9.3-p392/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => debug:graph
Here's what I came up with for dumping the graph
YAML. Technically:
Runner.new(@dataset, GraphBuilder.new.graph).calculate
# And than transform this to a YAML file probably :) but first:
Still when I try to execute this I get an error based on the fixtures
:
Refinery::IncalculableGraphError:
Insufficient data to calculate 1 items:
* NodeDemand for #<Refinery::Node key=:my_residence>
How to resolve this?
I'm not sure what to scale in order to create a full correct local dataset. I now only scale the attributes of a dataset which have the proportional: true
flag in the attributes. But I'm not sure that's all you need to scale.
This is assuming that you already scaled the exported demands
, max_demands
, etc. from the graph.yml
.
Related discussion: quintel/etmodel#2324 (comment)
It would be great if Atlas were to know if all the filenames of inputs are unique.
The rake import
task (original ticket: #30) chokes on the file .DS_Store
.
When I run rake import
on etsource
or atlas
, I get the following error message:
โ atlas git:(master) rake import
cd ../etsource
rake aborted!
File exists - data/datasets/.DS_Store
/Users/quintel/Projects/etsource/Rakefile:38:in `block (2 levels) in <top (required)>'
/Users/quintel/Projects/etsource/Rakefile:28:in `each'
/Users/quintel/Projects/etsource/Rakefile:28:in `block in <top (required)>'
Tasks: TOP => import
I can circumvent this, by specifying a country: rake import DATASET=...
.
I noticed that ZeroDisabledSectors only disables nodes based on their groups
but not based on their sector
. Additionally, metal_industry
is the only group that will ever get disabled.
Probably some more of the feature flags of Dataset
should lead to disabling of nodes based on their sector!?
Should init. inputs be a separate class InitializerInput
or should we continue using Input
?
And if separate, then should InitializerInput
and Input
share a common base class? Maybe that would be called Input
then and what is now Input
becomes UserInput
!?
There was some discussion about this in another issue already
As described here it should be possible to change the local dataset to the circumstances as they exist in that certain municipality/city/area. To do this we had the idea to adjust values using "initializer inputs" (the queries that live inside of, adjust_scaling/**/*.ad
files).
My question:
Should it be possible to overwrite anything in the graph?
We export the "sparse graph" (i.e. the graph before refinery calcultates the rest of the demands). Should it therefor only be possible to alter parts of the sparse graph? The upsides to this approach is that refinery will calculate a valid graph in the end (so, there won't be any calculation errors). The downside is that the level of control you have over the graph is entirely determined by refinery and demands might trickle down in a way that's not desired [? if that's even a thing].
When you can alter anything in the graph (assuming therefor that the graph is first fully calculated in Refinery and than these 'edits' are overwriting the values in the graph) the downside is that Refinery has a chance of not calculating correctly (i.e. demands might not add up in combination with the shares etc.). This can be fixed by expanding the validate task in Atlas and make sure that everyting runs as planned when these initial inputs are set (so set initial input of x, run rake validate <OPTIONS/>
and repeat). The upside is that you'd have more control over the graph. Than again you probably need to edit a lot of shares and nodes manually when changing the demand of a certain node in the middle of the fully calculated graph.
We're nearing the end of the stage0 phase of the local dataset project. The scaling is being reviewed as we speak and the basic implementation of the initializer inputs is in a pull request as well.
After both these pull requests are merged I think we need @DorinevanderVlies to test this locally on her machine to see if the workflow makes sense and if anything odd comes up.
Open todo (in somewhat prioritized order):
Refer to planning document
Running the new "bundle exec rake debug" task doesn't create/update the static.yml file. Is this a bug or a feature? How do I get etengine and model working again? @antw
a = Document.new(key: :a).save
b = Document.new(key: :b).save
expect { b.update_attributes!(key: :a) }.to raise_error
Running
rake "debug:subgraph[../etsource/data,transport]" --trace
gives me the following output:
rake "debug:subgraph[../etsource/data,transport]" --trace
** Invoke debug:subgraph (first_time)
** Execute debug:subgraph
Setting up graph structure...
Setting up Refinery graph...
rake aborted!
wrong number of arguments (2 for 1)
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/bundler/gems/refinery-f1ca52a644ee/lib/refinery/diagram/base.rb:30:in draw_to' /Users/kruip/Projects/atlas/lib/tasks/debug.rb:110:in
block (2 levels) in <top (required)>'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:in call' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:236:in
block in execute'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:in each' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:231:in
execute'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:175:in block in invoke_with_call_chain' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in
mon_synchronize'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:168:in invoke_with_call_chain' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/task.rb:161:in
invoke'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:149:in invoke_task' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in
block (2 levels) in top_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in each' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:106:in
block in top_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:115:in run_with_threads' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:100:in
top_level'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:78:in block in run' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:165:in
standard_exception_handling'
/Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/lib/rake/application.rb:75:in run' /Users/kruip/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rake-10.1.0/bin/rake:33:in
<top (required)>'
/Users/kruip/.rbenv/versions/1.9.3-p194/bin/rake:23:in load' /Users/kruip/.rbenv/versions/1.9.3-p194/bin/rake:23:in
When we tried to delete all the factors from the inputs (since they are like a 'wax nose'), we discovered that Tome does not write back the comments.
-# changed max value for slider that was set to the output of the CHP:
-# future:Q(max_percentage_heat_network_agriculture)
-# This was not good, as tapped heat can also be produced centrally.
-# This central production is now also included in the named query.
-
-- factor = 100.0
Currently, there are defined in ConverterGroups
sheet of the InputExcel. The xls2yaml script ports them to a yaml file that lives in etsource.
including @antw. Confer quintel/etengine#576.
From: #87 (comment)
How would you guys feel about naming these two new classes simply "Derived" and "Full"? "Dataset" already appears in the module tree, and it would shorten the .ad file names ("datasets/nl/nl.full.ad" instead of "datasets/nl/nl.full_dataset.ad").
For the sake of consistency should the rake scale
task also be taking FULL
and DERIVED
as arguments? Or should they still be called FULL_DATASET
. I personally think that the _DATASET
suffix provides some more context than just FULL
.
In this case, the CENTRAL_PRODUCTION(self.key)
could be used automatically, right @antw?
This ticket describe the functionality required to model the Industry for both The Netherlands and Germany.
1) industry_transformation_generic_coal
Part of the coal use in the Industry is used for transformation purposes. This is not counted in the final demand of coal, but result in primary energy requirements of coal. When the metal industry is switched on, this is modeled almost correctly for NL. However, when the metal industry will be switched off, this coal requirement need to be accounted for in the industry_transformation_generic_coal
converter by setting the output.coal
conversion (coal to final demand) and the output.loss
conversion (coal that to be modeled).
This efficiency is dependent on 1) whether or not the metal industry is specified, 2) the quality of the metal analysis.
See also https://github.com/quintel/rdr/issues/106 for the research background.
2) Blastfurnaces and cokesovens
The efficiencies of blastfurnaces and cokesovens are quite dependent on local circumstances. Therefore, e.g. input.coal
, input.cokes
and output.coal_gas
are calculated in the analysis and currently set in the respective converters. However, it is most likely that the efficiencies for Germany will differ from the efficiencies of The Netherlands. Also efficiencies between years can differ.
See also https://github.com/quintel/rdr/issues/107 for the research background.
A potential solution for the technical part of this problem (as @antw suggest in https://github.com/quintel/rdr/issues/64#issuecomment-21788427) is to query the efficiencies based on a table that is generated in a analysis that is output from a analysis, e.g. in case of industry_transformation_generic_coal
for NL 2011:
~ output.coal = EFF(โฆ, output.coal)
~ output.loss = EFF(โฆ, output.loss)
@ChaelKruip showed me the initial idea for creating local datasets (which was a picture on a whiteboard - in a circular movement) which was drawn by @antw
I remembered him telling me explicitly that a local dataset needs to be put in a single .ad
-file.
But I was wondering what the reasoning behind that is?
Wouldn't it be easier to just keep the data structure exactly the same to a non-local dataset?
Currently, there are still some problems with initializing the whole graph with solely final demands from the energy balance and related parent and child shares. To solve https://github.com/quintel/rdr/issues/105 we need to query a demand from a csv file.
A proposed syntax is ~ demand = DEMAND(filename, node_key)
. The values in the csv files will be in TJ. It would be nice if we can store the demand files in for example etsource/data/datasets/country/demands/...
.
The vast majority of slots in the graph have only a single attribute: the conversion
(called share
in Atlas/Refinery). However, a small number have an additional attribute: reset_to_zero
.
The Node documents provide no way for us to add additional attributes for a slot; you can only specify the conversion:
~ input.gas = 0.5
I'm unsure how to go about supporting extra attributes for slots. I have an idea, but it may not be necessary as the only extra attribute I can find is reset_to_zero
, and it appears to be used only on the four input slots for "coupling_carrier".
@dennisschoenmakers @ChaelKruip Instead of trying to shoehorn slot attributes into Atlas, do you think it would be reasonable to just have the ETEngine importer assign reset_to_zero: true
to all "coupling_carrier" input slots?
@ChaelKruip proposed an alternative idea for the initializer inputs and I want to test how feasible this is.
The current idea is to make initializer inputs behave like user_values
. So:
[ Atlas::GraphDeserializer / Atlas::Runner]
|
V
[ Atlas::FullExporter ]
|
V
[ Load data from dumped MessagePack ]
|
[ ... ]
|
V
[ Overwrite inputs with initializer inputs ] [?]
As an expansion/extra-step to that idea @ChaelKruip proposed the following:
O B
/
/ (A-B share: 0.5)
A (demand = 5) O
\ (A-C share: 0.5)
\
O C
Than this is the full-graph
O B (demand = 2.5)
/
/ (A-B share: 0.5)
A (demand = 5) O
\ (A-C share: 0.5)
\
O C (demand = 2.5)
If a researcher than knows the demand of node/converter B
and sets it to for example (10). This would work the same as the idea proposed above (as user_values
).
O B (demand = 10)
/
/ (A-B share: 0.5)
A (demand = 5) O
\ (A-C share: 0.5)
\
O C (demand = 2.5)
The extra step
Than the 'altered' sparse graph (if assuming the sparse graph will have the same starting points and edges) will look like:
O B
/
/ (A-B share: 0.5)
A (demand = 20) O
\ (A-C share: 0.5)
\
O C
Should this be reflected in the graph.yml
? Is that useful?
It would be useful - before running everything through ETEngine - to be able to check through git
which numbers will change.
Refinery completely ignores coupling_carrier when performing it's calculations โ a requirement for outputting a valid graph. Yet, ETEngine still requires that we provide a conversion for those slots.
As a temporary solution, I added the old InputExcel coupling carrier conversions, however these numbers are so specific I suspect they might be the result of some calculation in InputExcel rather than a hand-edited number.
@ChaelKruip Could you shed some light on how you'd like to handle these slots?
@dennisschoenmakers Please jump in if you have any opinions on this.
When I am running Atlas on the PL 2012 dataset (quintel/etsource@new_countries
) I get the following validation error:
โ atlas git:(master) rake debug DATASET=pl
Post-calculation validations failed with the following errors:
* #<Refinery::Slot (out, waste_mix) node=:energy_distribution_biogenic_waste> has an undetermined share
* #<Refinery::Slot (out, biogenic_waste) node=:energy_distribution_biogenic_waste> has an undetermined share
This is most likely cause by the zero share for biogenic waste in the energy_distribution_waste_mix_child_share file.
@antw Any idea to solve this validation error?
Output of tree etsource/datasets/example
datasets/example
โโโ central_producers.csv
โโโ demands
โย ย โโโ metal_demands.csv
โโโ efficiencies
โย ย โโโ energy_cokesoven_consumption_coal_gas_efficiency.csv
โย ย โโโ energy_cokesoven_transformation_coal_efficiency.csv
โย ย โโโ energy_mixer_for_gas_power_fuel_efficiency.csv
โย ย โโโ energy_steel_blastfurnace_bat_transformation_cokes_efficiency.csv
โย ย โโโ energy_steel_blastfurnace_current_transformation_cokes_efficiency.csv
โย ย โโโ industry_chemicals_fertilizers_steam_methane_reformer_hydrogen_efficiency.csv
โย ย โโโ industry_refinery_transformation_crude_oil_efficiency.csv
โย ย โโโ industry_steel_blastfurnace_bat_consumption_useable_heat_efficiency.csv
โย ย โโโ industry_steel_blastfurnace_current_consumption_useable_heat_efficiency.csv
โย ย โโโ industry_transformation_generic_coal_efficiency.csv
โโโ energy_balance.csv
โโโ energy_balance.gpg
โโโ example.ad
โโโ load_profiles
โย ย โโโ total_demand.csv
โโโ primary_production.csv
โโโ shares
โย ย โโโ agriculture_final_demand_electricity_parent_share.csv
โย ย โโโ buildings_final_demand_coal_parent_share.csv
โย ย โโโ buildings_final_demand_crude_oil_parent_share.csv
โย ย โโโ buildings_final_demand_electricity_parent_share.csv
โย ย โโโ buildings_final_demand_for_cooling_electricity_parent_share.csv
โย ย โโโ buildings_final_demand_for_lighting_electricity_parent_share.csv
โย ย โโโ buildings_final_demand_for_space_heating_electricity_parent_share.csv
โย ย โโโ buildings_final_demand_for_space_heating_network_gas_parent_share.csv
โย ย โโโ buildings_final_demand_network_gas_parent_share.csv
โย ย โโโ buildings_final_demand_wood_pellets_parent_share.csv
โย ย โโโ buildings_local_production_electricity_parent_share.csv
โย ย โโโ buildings_useful_demand_after_motion_detection_light_child_share.csv
โย ย โโโ buildings_useful_demand_cooling_child_share.csv
โย ย โโโ buildings_useful_demand_for_space_heating_child_share.csv
โย ย โโโ buildings_useful_demand_light_child_share.csv
โย ย โโโ energy_distribution_hydrogen_child_share.csv
โย ย โโโ energy_distribution_waste_mix_child_share.csv
โย ย โโโ energy_distribution_wood_parent_share.csv
โย ย โโโ energy_locally_available_hydrogen_before_p2g_child_share.csv
โย ย โโโ energy_national_gas_network_natural_gas_child_share.csv
โย ย โโโ energy_national_gas_network_natural_gas_greengas_child_share.csv
โย ย โโโ industry_aluminium_burner_network_gas_parent_share.csv
โย ย โโโ industry_final_demand_coal_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_coal_parent_share.csv
โย ย โโโ industry_final_demand_crude_oil_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_crude_oil_parent_share.csv
โย ย โโโ industry_final_demand_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_coal_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_coal_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_crude_oil_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_crude_oil_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_network_gas_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_network_gas_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_steam_hot_water_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_wood_pellets_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_chemical_wood_pellets_parent_share.csv
โย ย โโโ industry_final_demand_for_metal_coal_parent_share.csv
โย ย โโโ industry_final_demand_for_metal_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_metal_network_gas_parent_share.csv
โย ย โโโ industry_final_demand_for_metal_steam_hot_water_parent_share.csv
โย ย โโโ industry_final_demand_for_other_coal_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_other_coal_parent_share.csv
โย ย โโโ industry_final_demand_for_other_crude_oil_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_other_crude_oil_parent_share.csv
โย ย โโโ industry_final_demand_for_other_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_other_food_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_other_network_gas_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_other_network_gas_parent_share.csv
โย ย โโโ industry_final_demand_for_other_paper_electricity_parent_share.csv
โย ย โโโ industry_final_demand_for_other_steam_hot_water_parent_share.csv
โย ย โโโ industry_final_demand_for_other_wood_pellets_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_for_other_wood_pellets_parent_share.csv
โย ย โโโ industry_final_demand_network_gas_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_network_gas_parent_share.csv
โย ย โโโ industry_final_demand_steam_hot_water_parent_share.csv
โย ย โโโ industry_final_demand_wood_pellets_non_energetic_parent_share.csv
โย ย โโโ industry_final_demand_wood_pellets_parent_share.csv
โย ย โโโ industry_steel_blastfurnace_burner_coal_gas_parent_share.csv
โย ย โโโ industry_transformation_generic_coal_parent_share.csv
โย ย โโโ residences_final_demand_coal_parent_share.csv
โย ย โโโ residences_final_demand_crude_oil_parent_share.csv
โย ย โโโ residences_final_demand_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_appliances_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_cooking_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_cooling_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_hot_water_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_hot_water_network_gas_parent_share.csv
โย ย โโโ residences_final_demand_for_lighting_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_space_heating_electricity_parent_share.csv
โย ย โโโ residences_final_demand_for_space_heating_network_gas_parent_share.csv
โย ย โโโ residences_final_demand_network_gas_parent_share.csv
โย ย โโโ residences_final_demand_steam_hot_water_parent_share.csv
โย ย โโโ residences_final_demand_wood_pellets_parent_share.csv
โย ย โโโ residences_new_houses_useful_demand_for_cooling_child_share.csv
โย ย โโโ residences_new_houses_useful_demand_for_heating_child_share.csv
โย ย โโโ residences_old_houses_useful_demand_for_cooling_child_share.csv
โย ย โโโ residences_old_houses_useful_demand_for_heating_child_share.csv
โย ย โโโ residences_space_heater_micro_chp_network_gas_child_share.csv
โย ย โโโ residences_useful_demand_for_cooling_after_insulation_parent_share.csv
โย ย โโโ residences_useful_demand_for_hot_water_after_solar_heater_and_p2h_parent_share.csv
โย ย โโโ residences_useful_demand_for_hot_water_for_houses_with_p2h_child_share.csv
โย ย โโโ residences_useful_demand_for_hot_water_for_houses_with_solar_heater_child_share.csv
โย ย โโโ residences_useful_demand_for_space_heating_after_insulation_for_houses_with_solar_heater_child_share.csv
โย ย โโโ residences_useful_demand_for_space_heating_after_insulation_parent_share.csv
โย ย โโโ residences_water_heater_fuel_cell_chp_network_gas_child_share.csv
โย ย โโโ residences_water_heater_micro_chp_network_gas_child_share.csv
โย ย โโโ residences_water_heater_solar_thermal_parent_share.csv
โย ย โโโ transport_final_demand_bio_ethanol_parent_share.csv
โย ย โโโ transport_final_demand_biodiesel_parent_share.csv
โย ย โโโ transport_final_demand_diesel_parent_share.csv
โย ย โโโ transport_final_demand_electricity_parent_share.csv
โย ย โโโ transport_final_demand_for_road_electricity_parent_share.csv
โย ย โโโ transport_final_demand_for_road_greengas_parent_share.csv
โย ย โโโ transport_final_demand_for_road_natural_gas_parent_share.csv
โย ย โโโ transport_final_demand_gasoline_parent_share.csv
โย ย โโโ transport_road_mixer_compressed_network_gas_parent_share.csv
โย ย โโโ transport_road_mixer_diesel_mix_parent_share.csv
โย ย โโโ transport_road_mixer_gasoline_mix_parent_share.csv
โย ย โโโ transport_road_mixer_lng_child_share.csv
โย ย โโโ transport_shipping_mixer_diesel_child_share.csv
โย ย โโโ transport_shipping_mixer_lng_child_share.csv
โโโ time_curves
โโโ energy_distribution_woody_biomass_time_curve.csv
โโโ energy_extraction_coal_time_curve.csv
โโโ energy_extraction_crude_oil_time_curve.csv
โโโ energy_extraction_lignite_time_curve.csv
โโโ energy_extraction_natural_gas_time_curve.csv
โโโ energy_extraction_uranium_oxide_time_curve.csv
5 directories, 125 files
What exactly needs to be scaled and what doesn't need to be scaled? I heared that @jorisberkhout recalled that there was an issue about this explaining that.
Files that needs to be changed when automatically generating a dataset:
etsource/datasets.yml (add a row with which dataset there is)
mv etsource/datasets/<local_dastaset>/<source_dataset>.ad -> etsource/datasets/<local_dastaset>/<local_dataset>.ad
In an attempt to resolves the issues https://github.com/quintel/etdataset/issues/607 and quintel/etsource#921 (comment), I introduced and renamed several shares and updated the edges accordingly.
A little background
After we merged this pull request the P2H node plays a very similar role in fulfilling the hot water demand as the add-on heat pump does in fulfilling the space heating demand. I therefore tried to replicate the add-on heat pump situation for the P2H node. Below is a screenshot of the graph (on BETA) and the initial shares that are defined for this part of the graph including their values (space heating in RED, hot water in BLUE and the shared solar water heater in green).
The issue
The space heating part of the graph (RED) shown below contains enough information to initialise the graph (as it always did). Interestingly, the hot water part (BLUE) contains the same amount of information, but results in incalculable parts of the graph when I run rake debug
on Atlas. My guess was that this has to do with the righthand share of the blue part being 1-0
. I changed this to 0.999-0.001
and this indeed resulted in a fully initialised graph with correct results.
The question
@antw, do you know why the 1-0
share results in an incalculable graph and how we can best resolve this? I would like to resolve this before the September deploy.
NB
I have only updated the shares for the NL dataset as doing it for all is quite tedious work. Any failing specs associated with other datasets are thus to be expected and I will resolve those once this issue is resolved.
Including @ChaelKruip .
After running all Research Analyses, a bunch of CSV files are created on etdataset
that need to be copied to etsource
. It would be great if this could be done automagically....
I made a list of the old and new file names (including directories) in an Excel workbook.
Please be aware of the following:
etsource/data/datasets/<country>
folder, _except_ for the country.ad
fileSpecify:
number_of_residences
as an argument, and a country like nl
.The current way of "validating" a national dataset 'graph' is through rake debug
. However, I can imagine that you'd want something similar for a local dataset.
So you can do
Atlas::Dataset.create(properties)
Which calls new and save! Similar to ActiveRecord.
Atlas throws a really undescriptive error on this line, when somebody formats his or her query wrongly (cough, @RobTerwel). For instance
- query = EACH(
throws an error.
- query =
EACH(
Doesn't.
It would be practical if a little more descriptive error is thrown stating that you can't do this and in which file this error occurs.
Now it's sometimes a bit hard to read when all different kinds of converters are standing listed on top of each other, like import, production, export, conversions, etc.
I'm not exactly sure how to do this in an elegant way though...
After importing the graph I compared beta
and local
by performing the following code in the sandbox:
Hash[@scope.graph.converters.map{|c| [c.key, c.demand] }]
+--------------------------------------------------------+--------------------+--------------------+---------------------+
| key | beta | local | difference |
+--------------------------------------------------------+--------------------+--------------------+---------------------+
| energy_steel_blastfurnace_current_transformation_cokes | 89066589999.99962 | 0.0 | 89066589999.99962 |
| energy_distribution_cokes | 53608499999.99974 | 0.0 | 53608499999.99974 |
| energy_cokesoven_transformation_coal | 72821157188.91655 | 0.0 | 72821157188.91655 |
| energy_cokesoven_consumption_coal_gas | 7400469944.584353 | 0.0 | 7400469944.584353 |
| industry_locally_available_electricity | 141496247385.3905 | 141199400000.00006 | 296847385.3904419 |
| energy_distribution_coal_gas | 50610397143.576675 | 18299614029.199936 | 32310783114.37674 |
| energy_export_coal_gas | 25207160555.182816 | 0.0 | 25207160555.182816 |
| energy_import_coal_gas | 0.0 | 18299614029.199936 | -18299614029.199936 |
| energy_distribution_coal | 362651179619.91205 | 254371932430.99564 | 108279247188.91641 |
| energy_import_coal | 362651179619.91205 | 254371932430.99564 | 108279247188.91641 |
| energy_power_mv_distribution_network_electricity | 339955443308.01074 | 339658595922.6203 | 296847385.3904419 |
| energy_power_mv_transport_network_electricity | 328822721390.2025 | 328525874004.8121 | 296847385.3904419 |
| energy_national_gas_network_natural_gas | 1402481155592.2878 | 1402481155592.288 | -0.000244140625 |
| energy_power_transformer_mv_hv_electricity | 328822721390.2025 | 328525874004.8121 | 296847385.3904419 |
| energy_power_hv_network_electricity | 359615461390.20245 | 359304650492.1222 | 310810898.08026123 |
| energy_interconnector_imported_electricity | 65761861390.20233 | 65451050492.12213 | 310810898.0802002 |
| energy_import_electricity | 65761861390.20233 | 65451050492.12213 | 310810898.0802002 |
| energy_power_hv_network_shortage | 6.103515625e-05 | 0.0 | 6.103515625e-05 |
| energy_power_hv_network_loss | 16156109999.999962 | 16142146487.310165 | 13963512.689796448 |
| energy_export_network_gas | 1171278445596.1736 | 1171278445596.1733 | 0.000244140625 |
+--------------------------------------------------------+--------------------+--------------------+---------------------+
Some demands don't add up. I did filter out most rounding errors.
As specified here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.