Utility package for Sienna's simulation infrastructure
Home Page: https://nrel-sienna.github.io/InfrastructureSystems.jl/
License: BSD 3-Clause "New" or "Revised" License
The current transform_single_time_series does not support multiple resolutions. The code needs to allow for horizon to be expressed in terms of time (horizon * resolution) in addition to the current implementation.
julia> sys = PSB.build_system(PSB.SIIPExampleSystems, "5_bus_hydro_uc_sys")
[ Info: Deserializing with InMemoryTimeSeriesStorage is currently not supported. Using HDF
[ Info: Loaded time series from storage file existing=5_bus_hydro_uc_sys_time_series_storage.h5 new=/var/folders/27/2jr8c7gn4j72fvrg4qt81zrw8w_711/T/jl_PM3GKQ compression=CompressionSettings(false, CompressionTypes.DEFLATE = 1, 3, true)
System
======
System Units Base: DEVICE_BASE
Base Power: 100.0
Base Frequency: 60.0
Components
==========
Num components: 35
12×3 DataFrame
Row │ ConcreteType SuperTypes
│ String String
─────┼──────────────────────────────────────────────────────────────────
1 │ Arc Topology <: Component <: Infrast…
2 │ Area AggregationTopology <: Topology …
3 │ Bus Topology <: Component <: Infrast…
4 │ HydroDispatch HydroGen <: Generator <: StaticI…
5 │ HydroEnergyReservoir HydroGen <: Generator <: StaticI…
6 │ HydroPumpedStorage HydroGen <: Generator <: StaticI…
7 │ Line ACBranch <: Branch <: Device <: …
8 │ LoadZone AggregationTopology <: Topology …
9 │ PowerLoad StaticLoad <: ElectricLoad <: St…
10 │ TapTransformer ACBranch <: Branch <: Device <: …
11 │ ThermalStandard ThermalGen <: Generator <: Stati…
12 │ VariableReserve{ReserveUp} Reserve{ReserveUp} <: Service <:…
1 column omitted
TimeSeriesContainer
===================
Components with time series data: 7
Total StaticTimeSeries: 14
Total Forecasts: 14
Resolution: 60 minutes
First initial time: 2020-01-01T00:00:00
Last initial time: 2020-01-07T00:00:00
Horizon: 24
Interval: 1440 minutes
Forecast window count: 7
julia> re = RenewableDispatch(
"WindBusA",
true,
get_component(Bus, sys, "bus5"),
0.0,
0.0,
1.200,
PrimeMovers.WT,
(min = 0.0, max = 0.0),
1.0,
TwoPartCost(0.220, 0.0),
100.0,
)
┌ Warning: SystemUnitSetting not defined, using NATURAL_UNITS for displaying device specification.
└ @ PowerSystems ~/.julia/packages/PowerSystems/DdZ5W/src/utils/print.jl:69
WindBusA (RenewableDispatch):
name: WindBusA
available: true
bus: bus5 (Bus)
active_power: 0.0
reactive_power: 0.0
rating: 120.0
prime_mover: PrimeMovers.WT = 22
reactive_power_limits: (min = 0.0, max = 0.0)
power_factor: 1.0
operation_cost: TwoPartCost
base_power: 100.0
services: 0-element Vector{Service}
dynamic_injector: nothing
ext: Dict{String, Any}()
time_series_container: InfrastructureSystems.TimeSeriesContainer: 0
internal: InfrastructureSystems.InfrastructureSystemsInternal
julia> add_component!(sys, re)
julia> copy_time_series!(re, get_component(PowerLoad, sys, "bus2"))
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: H5A.c line 312 in H5Acreate2(): unable to create attribute
major: Attribute
minor: Unable to initialize object
#001: H5VLcallback.c line 1023 in H5VL_attr_create(): attribute create failed
major: Virtual Object Layer
minor: Unable to create file
#002: H5VLcallback.c line 989 in H5VL__attr_create(): attribute create failed
major: Virtual Object Layer
minor: Unable to create file
#003: H5VLnative_attr.c line 77 in H5VL__native_attr_create(): unable to create attribute
major: Attribute
minor: Unable to initialize object
#004: H5Aint.c line 166 in H5A__create(): attribute already exists
major: Attribute
minor: Object already exists
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: H5A.c line 885 in H5Aget_name(): not an attribute
major: Invalid arguments to routine
minor: Inappropriate type
ERROR: Error getting attribute name
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] h5a_get_name
@ ~/.julia/packages/HDF5/VJkAi/src/api.jl:207 [inlined]
[3] h5a_get_name(attr_id::HDF5.Group)
@ HDF5 ~/.julia/packages/HDF5/VJkAi/src/api_helpers.jl:27
[4] h5a_create
@ ~/.julia/packages/HDF5/VJkAi/src/api.jl:119 [inlined]
[5] create_attribute(parent::HDF5.Group, name::String, dtype::HDF5.Datatype, dspace::HDF5.Dataspace)
@ HDF5 ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:750
[6] create_attribute(parent::HDF5.Group, name::String, data::Bool; pv::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ HDF5 ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:1538
[7] create_attribute
@ ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:1535 [inlined]
[8] write_attribute(parent::HDF5.Group, name::String, data::Bool; pv::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ HDF5 ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:1560
[9] write_attribute
@ ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:1560 [inlined]
[10] setindex!
@ ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:882 [inlined]
[11] (::InfrastructureSystems.var"#116#117"{InfrastructureSystems.Hdf5TimeSeriesStorage, Base.UUID, String, String, String})(file::HDF5.File)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/fSRUY/src/hdf5_time_series_storage.jl:269
[12] h5open(::InfrastructureSystems.var"#116#117"{InfrastructureSystems.Hdf5TimeSeriesStorage, Base.UUID, String, String, String}, ::String, ::Vararg{String, N} where N; swmr::Bool, pv::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ HDF5 ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:458
[13] h5open
@ ~/.julia/packages/HDF5/VJkAi/src/HDF5.jl:456 [inlined]
[14] add_time_series_reference!(storage::InfrastructureSystems.Hdf5TimeSeriesStorage, component_uuid::Base.UUID, name::String, ts_uuid::Base.UUID)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/fSRUY/src/hdf5_time_series_storage.jl:266
[15] copy_time_series!(dst::RenewableDispatch, src::PowerLoad; name_mapping::Nothing, scaling_factor_multiplier_mapping::Nothing)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/fSRUY/src/component.jl:444
[16] #copy_time_series!#156
@ ~/.julia/packages/PowerSystems/DdZ5W/src/base.jl:1014 [inlined]
[17] copy_time_series!(dst::RenewableDispatch, src::PowerLoad)
@ PowerSystems ~/.julia/packages/PowerSystems/DdZ5W/src/base.jl:1014
[18] top-level scope
@ REPL[22]:1Stacktrace:
ERROR: LoadError: LoadError: MethodError: no method matching dirname(::Nothing)
Closest candidates are:
dirname(::AbstractString) at path.jl:161
Stacktrace:
[1] top-level scope
@ ~/.julia/packages/InfrastructureSystems/fbqgd/src/utils/logging.jl:20
[2] include(mod::Module, _path::String)
@ Base ./Base.jl:386
[3] include(x::String)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/fbqgd/src/InfrastructureSystems.jl:3
[4] top-level scope
@ ~/.julia/packages/InfrastructureSystems/fbqgd/src/InfrastructureSystems.jl:73
[5] include
@ ./Base.jl:386 [inlined]
[6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1213
[7] top-level scope
@ none:1
[8] eval
@ ./boot.jl:360 [inlined]
[9] eval(x::Expr)
@ Base.MainInclude ./client.jl:446
[10] top-level scope
@ none:1
in expression starting at /Users/raphaelsaavedra/.julia/packages/InfrastructureSystems/fbqgd/src/utils/logging.jl:20
in expression starting at /Users/raphaelsaavedra/.julia/packages/InfrastructureSystems/fbqgd/src/InfrastructureSystems.jl:3
ERROR: LoadError: Failed to precompile InfrastructureSystems [2cd47ed4-ca9b-11e9-27f2-ab636a7671f1] to /Users/raphaelsaavedra/.julia/compiled/v1.6/InfrastructureSystems/jl_lTGqWJ.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)
@ Base ./loading.jl:1360
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1306
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1021
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:914
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:901
[7] include
@ ./Base.jl:386 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1213
[9] top-level scope
@ none:1
[10] eval
@ ./boot.jl:360 [inlined]
[11] eval(x::Expr)
@ Base.MainInclude ./client.jl:446
[12] top-level scope
@ none:1
in expression starting at /Users/raphaelsaavedra/.julia/packages/PowerSystems/DdZ5W/src/PowerSystems.jl:3
ERROR: LoadError: Failed to precompile PowerSystems [bcd98974-b02a-5e2f-9ee0-a103f5c450dd] to /Users/raphaelsaavedra/.julia/compiled/v1.6/PowerSystems/jl_Oes5V7.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY)
@ Base ./loading.jl:1360
[3] compilecache(pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1306
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1021
[5] require(uuidkey::Base.PkgId)
@ Base ./loading.jl:914
[6] require(into::Module, mod::Symbol)
@ Base ./loading.jl:901
[7] include
@ ./Base.jl:386 [inlined]
[8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
@ Base ./loading.jl:1213
[9] top-level scope
@ none:1
[10] eval
@ ./boot.jl:360 [inlined]
[11] eval(x::Expr)
@ Base.MainInclude ./client.jl:446
[12] top-level scope
@ none:1
in expression starting at ...
I'm unable to run the following:
sort(
get_components(PowerLoad, sys),
by = load -> mean(values(get_time_series_array(Deterministic, load, "max_active_power")))
)The workaround is to wrap get_components(PowerLoad, sys) with collect(...).
These methods can be removed and change the requirement in PowerSimulations.jl
I think there is something wrong with the System display. This line results in the following error:
julia> c_sys14 = build_system("c_sys14")
[ Info: Deserializing with InMemoryTimeSeriesStorage is currently not supported. Using HDF
[ Info: Loaded time series from storage file existing=sys_time_series_storage.h5 new=/var/folders/27/2jr8c7gn4j72fvrg4qt81zrw8w_711/T/jl_7tpaXk
┌ Warning: Invalid range
│ valid_info.struct_name = "Bus"
│ field_name = "magnitude"
│ field_value = 1.07
│ valid_range = "voltage_limits"
│ valid_info.ist_struct =
│ Bus 6 (Bus):
│ number: 6
│ name: Bus 6
│ bustype: PV
│ angle: -0.24818581963359368
│ magnitude: 1.07
│ voltage_limits: (min = 0.94, max = 1.06)
│ base_voltage: 13.8
│ area: nothing
│ load_zone: nothing
│ ext: Dict{String,Any}()
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/flYfH/src/validation.jl:217
┌ Warning: Invalid range
│ valid_info.struct_name = "Bus"
│ field_name = "magnitude"
│ field_value = 1.062
│ valid_range = "voltage_limits"
│ valid_info.ist_struct =
│ Bus 7 (Bus):
│ number: 7
│ name: Bus 7
│ bustype: PQ
│ angle: -0.23335052099164186
│ magnitude: 1.062
│ voltage_limits: (min = 0.94, max = 1.06)
│ base_voltage: 13.8
│ area: nothing
│ load_zone: nothing
│ ext: Dict{String,Any}()
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/flYfH/src/validation.jl:217
┌ Warning: Invalid range
│ valid_info.struct_name = "Bus"
│ field_name = "magnitude"
│ field_value = 1.09
│ valid_range = "voltage_limits"
│ valid_info.ist_struct =
│ Bus 8 (Bus):
│ number: 8
│ name: Bus 8
│ bustype: PV
│ angle: -0.2331759880664424
│ magnitude: 1.09
│ voltage_limits: (min = 0.94, max = 1.06)
│ base_voltage: 18.0
│ area: nothing
│ load_zone: nothing
│ ext: Dict{String,Any}()
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/flYfH/src/validation.jl:217
┌ Warning: Invalid range
│ valid_info.struct_name = "ThermalStandard"
│ field_name = "reactive_power"
│ field_value = -0.169
│ valid_range = "reactive_power_limits"
│ valid_info.ist_struct =
│ Bus1 (ThermalStandard):
│ name: Bus1
│ available: true
│ status: true
│ bus: Bus 1 (Bus)
│ active_power: 2.0
│ reactive_power: -0.169
│ rating: 2.324
│ active_power_limits: (min = 0.0, max = 3.332)
│ reactive_power_limits: (min = 0.0, max = 0.1)
│ ramp_limits: nothing
│ operation_cost: ThreePartCost
│ base_power: 100.0
│ time_limits: nothing
│ prime_mover: ST
│ fuel: COAL
│ services: 0-element Array{Service,1}
│ time_at_status: 1.0e6
│ dynamic_injector: nothing
│ ext: Dict{String,Any}()
│ time_series_container: InfrastructureSystems.TimeSeriesContainer: 0
└ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/flYfH/src/validation.jl:217
System
======
System Units Base: SYSTEM_BASE
Base Power: 100.0
Base Frequency: 60.0
Components
==========
Num components: 70
7×3 DataFrame. Omitted printing of 2 columns
│ Row │ ConcreteType │
│ │ String │
├─────┼─────────────────┤
│ 1 │ Arc │
│ 2 │ Bus │
│ 3 │ Line │
│ 4 │ PowerLoad │
│ 5 │ TapTransformer │
│ 6 │ ThermalStandard │
│ 7 │ Transformer2W │
TimeSeriesContainer
===================
Components with time series data: 11
Total StaticTimeSeries: 0
Total Forecasts: 11
Resolution: 60 minutes
Error showing value of type System:
ERROR: AssertionError: initial_timestamp=2024-01-01T00:00:00 interval=0 seconds count=1
Stacktrace:
[1] get_initial_times(::DateTime, ::Int64, ::Second) at /Users/cbarrows/.julia/packages/InfrastructureSystems/flYfH/src/utils/utils.jl:373
[2] get_forecast_initial_times(::InfrastructureSystems.ForecastParameters) at /Users/cbarrows/.julia/packages/InfrastructureSystems/flYfH/src/time_series_parameters.jl:36
[3] get_forecast_initial_times at /Users/cbarrows/.julia/packages/InfrastructureSystems/flYfH/src/time_series_parameters.jl:185 [inlined]
[4] get_forecast_initial_times at /Users/cbarrows/.julia/packages/InfrastructureSystems/flYfH/src/system_data.jl:536 [inlined]
[5] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::InfrastructureSystems.SystemData) at /Users/cbarrows/.julia/packages/InfrastructureSystems/flYfH/src/utils/print.jl:86
[6] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::System) at /Users/cbarrows/.julia/packages/PowerSystems/OGkfl/src/utils/print.jl:18
[7] display(::REPL.REPLDisplay, ::MIME{Symbol("text/plain")}, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:137
[8] display(::REPL.REPLDisplay, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:141
[9] display(::Any) at ./multimedia.jl:323
[10] #invokelatest#1 at ./essentials.jl:712 [inlined]
[11] invokelatest at ./essentials.jl:711 [inlined]
[12] print_response(::IO, ::Any, ::Bool, ::Bool, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:161
[13] print_response(::REPL.AbstractREPL, ::Any, ::Bool, ::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:146
[14] (::REPL.var"#do_respond#38"{Bool,REPL.var"#48#57"{REPL.LineEditREPL,REPL.REPLHistoryProvider},REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:729
[15] #invokelatest#1 at ./essentials.jl:712 [inlined]
[16] invokelatest at ./essentials.jl:711 [inlined]
[17] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:2354
[18] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:1055
[19] run_repl(::REPL.AbstractREPL, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:206
[20] (::Base.var"#764#766"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:383
[21] #invokelatest#1 at ./essentials.jl:712 [inlined]
[22] invokelatest at ./essentials.jl:711 [inlined]
[23] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:367
[24] exec_options(::Base.JLOptions) at ./client.jl:305
[25] _start() at ./client.jl:484REPL print of empty system broken
MWE
julia> test_sys = System(100.0; time_series_in_memory = true)output:
Error showing value of type System:
ERROR: ArgumentError: type does not have a definite number of fields
Stacktrace:
[1] fieldcount at ./reflection.jl:725 [inlined]
[2] buildcolumns at /Users/jdlara/.julia/packages/Tables/iG2a3/src/fallbacks.jl:206 [inlined]
[3] columns at /Users/jdlara/.julia/packages/Tables/iG2a3/src/fallbacks.jl:262 [inlined]
[4] DataFrame(::Array{Any,1}; copycols::Bool) at /Users/jdlara/.julia/packages/DataFrames/Zx5mm/src/other/tables.jl:55
[5] DataFrame at /Users/jdlara/.julia/packages/DataFrames/Zx5mm/src/other/tables.jl:46 [inlined]
[6] create_components_df(::InfrastructureSystems.Components) at /Users/jdlara/.julia/dev/InfrastructureSystems/src/utils/print.jl:204
[7] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::InfrastructureSystems.Components) at /Users/jdlara/.julia/dev/InfrastructureSystems/src/utils/print.jl:27
[8] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::InfrastructureSystems.SystemData) at /Users/jdlara/.julia/dev/InfrastructureSystems/src/utils/print.jl:70
[9] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::System) at /Users/jdlara/.julia/dev/PowerSystems/src/utils/print.jl:18
[10] display(::REPL.REPLDisplay{REPL.LineEditREPL}, ::MIME{Symbol("text/plain")}, ::System) at /Users/jdlara/.julia/packages/OhMyREPL/07uNa/src/output_prompt_overwrite.jl:8
[11] display(::REPL.REPLDisplay, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:218
[12] display(::Any) at ./multimedia.jl:328
[13] #invokelatest#1 at ./essentials.jl:710 [inlined]
[14] invokelatest at ./essentials.jl:709 [inlined]
[15] print_response(::IO, ::Any, ::Bool, ::Bool, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:238
[16] print_response(::REPL.AbstractREPL, ::Any, ::Bool, ::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:223
[17] (::REPL.var"#do_respond#54"{Bool,Bool,REPL.var"#64#73"{REPL.LineEditREPL,REPL.REPLHistoryProvider},REPL.LineEditREPL,REPL.LineEdit.Prompt})(::Any, ::Any, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:822
[18] #invokelatest#1 at ./essentials.jl:710 [inlined]
[19] invokelatest at ./essentials.jl:709 [inlined]
[20] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/LineEdit.jl:2355
[21] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:1144
[22] (::REPL.var"#38#42"{REPL.LineEditREPL,REPL.REPLBackendRef})() at ./task.jl:356When using to_json(xxx; force = true) the file names that check for removal aren't the same as the ones used to write the files leading to them not being deleted.
Some simulation status recorder messages are not flushed to file for long periods of time. Users want these to be flushed so that they can tail the log file and see current status. We could allow each recorder to be flushed immediately.
I am about to submit a PR to add FunctionData, a better way to represent cost functions, to InfrastructureSystems. I will have adapted the existing HDF serialization/deserialization code for certain types of FunctionData, but FunctionData can represent more cost functions than the existing data structures, so more serialization/deserialization routines will need to be implemented in the future — for instance, something to handle polynomials of arbitrary degree, which might look like
function transform_array_for_hdf(data::Vector{PolynomialFunctionData})
coeffs = data .|> get_coefficients .|> collect
return cat((hcat(([a, b] for (a, b) in coeff)...) for coeff in coeffs)...; dims=3)
end
This would also be a good opportunity to clean up the code duplication that exists between the various methods of transform_array_for_hdf, etc.
I have disabled the ambiguity tests in IS runtests.jl but these are the detected ones.
Ambiguity #1
deserialize(::Type{T}, data::Dict) where T<:InfrastructureSystems.InfrastructureSystemsType in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/serialization.jl:132
deserialize(::Type{InfrastructureSystems.SystemData}, raw; time_series_read_only, time_series_directory) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/system_data.jl:555
Ambiguity #2
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), forecast::InfrastructureSystems.Forecast, index::Int64) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), f::InfrastructureSystems.Scenarios, initial_time) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/scenarios.jl:211
Ambiguity #3
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), f::InfrastructureSystems.Deterministic, initial_time) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/deterministic.jl:258
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), forecast::InfrastructureSystems.Forecast, index::Int64) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
Ambiguity #4
get_window(forecast::InfrastructureSystems.Forecast, index::Int64; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
get_window(f::InfrastructureSystems.Scenarios, initial_time; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/scenarios.jl:211
Ambiguity #5
get_window(forecast::InfrastructureSystems.Forecast, index::Int64; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
get_window(f::InfrastructureSystems.Probabilistic, initial_time; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/probabilistic.jl:259
Ambiguity #6
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), forecast::InfrastructureSystems.Forecast, index::Int64) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
(::InfrastructureSystems.var"#get_window##kw")(::Any, ::typeof(InfrastructureSystems.get_window), f::InfrastructureSystems.Probabilistic, initial_time) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/probabilistic.jl:259
Ambiguity #7
get_window(f::InfrastructureSystems.Deterministic, initial_time; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/deterministic.jl:258
get_window(forecast::InfrastructureSystems.Forecast, index::Int64; len) in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/forecasts.jl:64
Ambiguity #8
check_limits(::Type{T}, valid_info::InfrastructureSystems.ValidationInfo, field_value) where T<:Union{Nothing, Real} in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/validation.jl:175
check_limits(::Type{T}, valid_info::InfrastructureSystems.ValidationInfo, field_value) where T<:Union{Nothing, NamedTuple} in InfrastructureSystems at /Users/jdlara/.julia/dev/InfrastructureSystems/src/validation.jl:184
Pretty tables 2.0 no longer supports passing the kwarg filters_col this is impacting IS.show_recorder_events
julia> using PowerSystems
[ Info: Precompiling PowerSystems [bcd98974-b02a-5e2f-9ee0-a103f5c450dd]
ERROR: LoadError: LoadError: UndefVarError: convert_type not defined
specifically
convert_type: v0.15.4...v0.15.5#diff-814b9e4969b1650fb0ca1da000c6fa05L161)convert_type e.g. https://github.com/NREL-SIIP/PowerSystems.jl/blob/0fb1f4b5e2c686b29ca828f57347899ef445e9b8/src/base.jl#L1169If that's right, here's advice on what to do https://github.com/SciML/ColPrac/blob/master/README.md#accidental-breaking-releases
Aside: Perhaps CI on the InfraSys repo should have run PSys tests? Here's how ChainRules.jl runs Zygote.jl tests for PR 🙂
https://github.com/JuliaDiff/ChainRules.jl/blob/9663f935cecbdf490525f58d9adf6616340f7510/.github/workflows/IntegrationTestZygote.yml
If IS.convert_type was private, then PowerSys should not have been using it.
Or if IS.convert_type was public API, ISv0.15.5 should have been v0.16.0?
FWIW, i think the best way forward it to release a v0.15.6 that's identical to v0.15.4, then re-relase what was v0.15.5 as v0.16.0
In the current re-organization a lot of the testing for the optimization submodule is implemented in PowerSimulations.jl particularly the pieces related to the results.
Coverage of this code directly in InfrastructureSystems.jl needs to be increased.
allow to run add_components and take as an argument and iterator of components.
We currently only support adding supplemental attributes one at a time. This is likely to be slow if there are thousands of attributes because each addition causes one row to be added to an SQLite table. We support bulk addition of time series for the same reason and could do the same for supplemental attributes.
This is a list of TODOs that remain unfinished from #344, they should all be addressed before IS is released.
retransform_hdf_array, use of map rather than preallocation might be a performance issue (see
etc.).transform_array_for_hdf, see review comment https://github.com/NREL-Sienna/InfrastructureSystems.jl/pull/344/files/241514d1b15282f1259f9a780b2c62a8fb4c6238#r1567689474convert_data as per #344 (comment)These seem to all be potential performance issues.
After the change to the multi logger. This function is triggering an error in PSID.
We could remove the scoped_enum macro from this package and use EnumX.@enumx instead. We would need to replace all usages across PowerSystems.jl and PowerSimulations.jl.
in InfrastructureSystems v1.4.3, a keyword to add_component! was renamed from deserialization_in_progress -> allow_existing_time_series
https://github.com/NREL-SIIP/InfrastructureSystems.jl/pull/220/files#r625452331
AFAICT add_component! is public API, so this is a breaking change.
In particular this breaks PowerSystems.jl; it causes a MethodError at https://github.com/NREL-SIIP/PowerSystems.jl/blob/v1.4.2/src/base.jl#L383-L386
Which in turn breaks our internal code (which simply calls add_component!(system, bus)), leading to PowerSystem's hitting the MethodError:
MethodError: no method matching add_component!(::InfrastructureSystems.Components, ::Bus; deserialization_in_progress=false, skip_validation=true)
Closest candidates are:
add_component!(::InfrastructureSystems.Components, ::T; skip_validation, allow_existing_time_series) where T<:InfrastructureSystems.InfrastructureSystemsComponent at /root/.julia/packages/InfrastructureSystems/q6GNU/src/components.jl:34 got unsupported keyword argument "deserialization_in_progress"
add_component!(!Matched::InfrastructureSystems.SystemData, ::Any; kwargs...) at /root/.julia/packages/InfrastructureSystems/q6GNU/src/system_data.jl:602
Stacktrace:
[1] kwerr(::NamedTuple{(:deserialization_in_progress, :skip_validation), Tuple{Bool, Bool}}, ::Function, ::InfrastructureSystems.Components, ::Bus)
@ Base ./error.jl:157
[2] add_component!(data::InfrastructureSystems.SystemData, component::Bus; kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol, Symbol}, NamedTuple{(:deserialization_in_progress, :skip_validation), Tuple{Bool, Bool}}})
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/q6GNU/src/system_data.jl:602
[3] add_component!(sys::System, component::Bus; skip_validation::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ PowerSystems ~/.julia/packages/PowerSystems/RdpU1/src/base.jl:383
[4] add_component!
@ ~/.julia/packages/PowerSystems/RdpU1/src/base.jl:369 [inlined]This will happen before PSY 4.0 is complete.
When scoped enums are used in print statements they print the whole module. For example
[ Info: Unit System changed to InfrastructureSystems.UnitSystemModule.UnitSystem.SYSTEM_BASE = 0Currently keys is not defined for InfrastructureSystems.FlattenIteratorWrapper.
julia> create_graph(sys)
ERROR: MethodError: no method matching keys(::InfrastructureSystems.FlattenIteratorWrapper{Bus})
Closest candidates are:
keys(::Union{Tables.AbstractColumns, Tables.AbstractRow}) at ~/.julia/packages/Tables/M26tI/src/Tables.jl:184
keys(::Union{HDF5.File, HDF5.Group}) at ~/.julia/packages/HDF5/pIJra/src/HDF5.jl:943
...
This prevents users from using functions like findall(x -> x.name === get_from(get_arc(branch)).name, buses) where buses is a InfrastructureSystems.FlattenIteratorWrapper.
Workaround is to use findall(x -> x.name === get_from(get_arc(branch)).name, collect(buses)) at the moment.
we need a method similar to get_forecast_initial_timestamp(sys) but for systems with only SingleTimeSeries
On a transformed system, the function get_forecast_initial_times returns an empty Array. This also breaks the printing of a transformed system because of line 94 in InfrastructureSystems/src/utils/print.jl
julia> get_forecast_initial_times(sys_RT)
Any[]When transforming a SingleTimeSeries into a Horizon = 1 forecast, getting the horizon resolution fails
This is the line with the issue.
In src/supplemental_constructors.jl on the master branch, one of each of the Deterministic, Probabilistic, and ScenarioBased (those defined at lines 18, 73 and 148, respectively) create timestamps for a TimeArray object with something like
initial_time:Dates.Hour(1):(initial_time + resolution * (time_steps - 1))
resulting in an error in the creation of the TimeArray due to a dimension mismatch between the length of the timestamp vector and the length of the data.
The intent seems to be to create the timestamps with something like
initial_time:resolution:(initial_time + resolution * (time_steps - 1))
I don't love how we special-case that Deterministic, a concrete type, sometimes also refers to DeterministicSingleTimeSeries, a different concrete type. I don't know about the typical user, but this definitely confused me before it was explained to me. We would have to think about how to resolve this without breaking existing code. For now we'll keep it as-is but at some point restore the behavior that you can pass abstract types to get_time_series; when that is done, there is a change that can be reverted in PSY. See #349 (comment). Broken out of #356.
If a component doesn't have the field name, IS throws an error when showing in the REPL.
In this example, CombinedVIwithVZ is a custom-defined component in LITS.jl
ERROR: type CombinedVIwithVZ has no field name
Stacktrace:
[1] getproperty(::Any, ::Symbol) at ./Base.jl:20
[2] get_name(::CombinedVIwithVZ) at /Users/jdlara/.julia/packages/InfrastructureSystems/VUeSS/src/InfrastructureSystems.jl:16
[3] summary(::CombinedVIwithVZ) at /Users/jdlara/.julia/packages/InfrastructureSystems/VUeSS/src/utils/print.jl:136
[4] show(::IOContext{REPL.Terminals.TTYTerminal}, ::MIME{Symbol("text/plain")}, ::CombinedVIwithVZ) at /Users/jdlara/.julia/packages/InfrastructureSystems/VUeSS/src/utils/print.jl:140
[5] display(::REPL.REPLDisplay, ::MIME{Symbol("text/plain")}, ::Any) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:132
[6] display(::REPL.REPLDisplay, ::Any) at /Users/sabae/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.2/REPL/src/REPL.jl:136This is a list of optimization related objects defined in PowerSimulations.jl that might make sense to move to IS.Optimization.
This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
Here are a few issues that came up in #349 that didn't quite merit being dealt with there but should probably be addressed at some point. Some of these might be spun off into their own issues.
resolution to the difference between the first two timestamps. This sort of validation should instead be done in the time series constructor, and it should compare the differences between all consecutive timestamps, so it is never possible to construct an invalid time series. More radically, one might ask why we are storing this redundant information at all rather than having a getter that computes it. See #349 (comment) and #349 (comment)check_consistency and allow the user to override. See #349 (comment)Dates.DateTime(0) as a null value, which could be problematic. This existed in the codebase before this PR so we haven't changed it yet. See #349 (comment)Deterministic, a concrete type, sometimes also refers to DeterministicSingleTimeSeries, a different concrete type. I don't know about the typical user, but this definitely confused me before it was explained to me. We would have to think about how to resolve this without breaking existing code. For now we'll keep it as-is but at some point restore the behavior that you can pass abstract types to get_time_series; when that is done, there is a change that can be reverted in PSY. See #349 (comment)currently:
IS.get_components(T, sys.data, filter_func)
correct
IS.get_components(filter_func, T, sys.data)
I'm opening the issue to track a feature request.
Currently the time series cache detects reading data at the same time stamp as "arbitrary access". In the original scope we didn't expect to have to read the same time-stamp twice but for the developments in PowerSimulationsDecomposition.jl this will be a necessary feature.
This line should be:
show(io, df, allrows=true, allcols=true)
TOML is part of the standard library starting in Julia 1.6.
Another catch-all issue for minor, low-priority refactors. There are some things we currently do using string manipulation, manually keeping track of lists of things, etc. that I think could be more elegantly accomplished using Julia's abilities to operate on itself. I'll start a list here:
Apply the same principle as PowerSystems and PowerSimulations
When serializing/deserializing a system there is a possibility to have the UUID clashes. This happens when the user modifies a deserialized system, serializes, and de-serializes back.
The attached file is a system JSON that I made with that workflow (github only allows certain extensions to be added to issues so I changed the extension to log)
Check UUID 281b8f3f-a231-4660-937b-6d0eb7f3f682. It is assigned to ThermalMultiStart DEER_PARK_ENERGY_CENTER_CC5 and also Bus MISSION 1
Also UUID 9732a557-2700-41c0-b5a7-d7147f8c98ea is assigned to SPENCER_STG_U4 and to BUS WICHITA FALLS 2
I won't add more instances of this situation happening, but as of this 3rd edit of the issue I have had 5 instances of repeated UUIDs
We are currently serializing string versions of time series data shapes to HDF5 attributes. This is causing problems with the new PRs from @GabrielKS. We can instead embed these eltypes into the metadata structs and handle their serialization/de-serialization in the normal way.
see https://travis-ci.org/NREL/PowerSystems.jl/jobs/614313755#L348
The characters <: are being rendered as <
I'm trying to update some private code from PowerSystems.jl v0.28 to v1
Below is using PowerSystems v1.3.0 and InfrastructureSystems v1.3.6
I'm testing with a 3 bus system, which should have SingleTimeSeries for each of the PowerLoad and ThermalStandard
julia> sys
System
======
System Units Base: SYSTEM_BASE
Base Power: 100.0
Base Frequency: 60.0
Components
==========
Num components: 26
8×3 DataFrame
Row │ ConcreteType SuperTypes Count
│ String String Int64
─────┼─────────────────────────────────────────────────────────────────────────
1 │ Arc Topology <: Component <: Infrast… 3
2 │ Bus Topology <: Component <: Infrast… 3
3 │ Line ACBranch <: Branch <: Device <: … 3
4 │ PhaseShiftingTransformer ACBranch <: Branch <: Device <: … 1
5 │ PowerLoad StaticLoad <: ElectricLoad <: St… 2
6 │ StaticReserveGroup{ReserveUp} Service <: Component <: Infrastr… 3
7 │ ThermalStandard ThermalGen <: Generator <: Stati… 2
8 │ VariableReserve{ReserveUp} Reserve{ReserveUp} <: Service <:… 9
TimeSeriesContainer
===================
Components with time series data: 4
Total StaticTimeSeries: 4
Total Forecasts: 0
Resolution: 60 minutesI convert these time-series to forecasts, with transform_single_time_series!, and that seems to work (no error or other output is shown)
julia> transform_single_time_series!(sys, 24, Hour(1))but when I then see an error upon trying to show the resulting system:
julia> sys
System
======
System Units Base: SYSTEM_BASE
Base Power: 100.0
Base Frequency: 60.0
Components
==========
Num components: 26
8×3 DataFrame
Row │ ConcreteType SuperTypes Count
│ String String Int64
─────┼─────────────────────────────────────────────────────────────────────────
1 │ Arc Topology <: Component <: Infrast… 3
2 │ Bus Topology <: Component <: Infrast… 3
3 │ Line ACBranch <: Branch <: Device <: … 3
4 │ PhaseShiftingTransformer ACBranch <: Branch <: Device <: … 1
5 │ PowerLoad StaticLoad <: ElectricLoad <: St… 2
6 │ StaticReserveGroup{ReserveUp} Service <: Component <: Infrastr… 3
7 │ ThermalStandard ThermalGen <: Generator <: Stati… 2
8 │ VariableReserve{ReserveUp} Reserve{ReserveUp} <: Service <:… 9
TimeSeriesContainer
===================
Components with time series data: 4
Total StaticTimeSeries: 4
Total Forecasts: 4
Resolution: 60 minutes
Error showing value of type System:
ERROR: AssertionError: 1 hour == 0 seconds
Stacktrace:
[1] macro expansion
@ ~/.julia/packages/InfrastructureSystems/v75Hd/src/utils/assert_op.jl:34 [inlined]
[2] get_initial_times(initial_timestamp::DateTime, count::Int64, interval::Hour)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/v75Hd/src/utils/utils.jl:404
[3] get_forecast_initial_times(params::InfrastructureSystems.ForecastParameters)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/v75Hd/src/time_series_parameters.jl:67
[4] get_forecast_initial_times
@ ~/.julia/packages/InfrastructureSystems/v75Hd/src/time_series_parameters.jl:232 [inlined]
[5] get_forecast_initial_times
@ ~/.julia/packages/InfrastructureSystems/v75Hd/src/system_data.jl:601 [inlined]
[6] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, data::InfrastructureSystems.SystemData)
@ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/v75Hd/src/utils/print.jl:93
[7] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, sys::System)
@ PowerSystems ~/.julia/packages/PowerSystems/yTZiA/src/utils/print.jl:18
[8] (::REPL.var"#38#39"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)Seemingly from an AssertionError in get_forecast_initial_times(sys)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.