Hi,
I have enabled schedule for sending the FogLAMP readings to North (Pi Server) and have set Producer Token and Ingress URL as shown in Pi Data Collection Manager as follows:

curl -s http://localhost:8081/foglamp/schedule/2b614d26-760f-11e7-b5a5-be2e44b06b34 | jq
{
"day": null,
"id": "2b614d26-760f-11e7-b5a5-be2e44b06b34",
"enabled": true,
"time": 0,
"repeat": 30,
"type": "INTERVAL",
"exclusive": true,
"processName": "North Readings to PI",
"name": "OMF to PI north"
}

curl -X GET http://localhost:8081/foglamp/category/SEND_PR_1 | jq
{
"applyFilter": {
.......

"URL": {
"type": "string",
"value": "https://ibmx3530m3-52-20:5460/ingress/messages",
"default": "https://pi-server:5460/ingress/messages",
"description": "The URL of the PI Connector to send data to"
},

"producerToken": {
"type": "string",
"value": "uid=3d2e6c31-746e-4bd9-8ada-cc8e286b441a&crt=20180716235212830&sig=6asBQ1ceQZAMsHPOmdbZb0stGz2Paye4KEUzuXN3/eQ=",
"default": "omf_north_0001",
"description": "The producer token that represents this FogLAMP stream"
},
.....

}
Yet, I do not see anything on Pi Asset Framework Ingested.
I looked up the /var/log/syslog and found these error messages - please help me resolve the problem.

Error Message from syslog:
WARNING: omf: omf_north_1: an error occurred during the request to the destination - error details |HTTPSConnectionPool(host='ibmx3530m3-52-20', port=5460): Max retries exceeded with url: /ingress/messages (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff276fe33c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))|
Jul 16 13:11:52 scspa0501709001 FogLAMP[31799] ERROR: omf: omf_north_1: cannot complete the sending operation - error details |an error occurred during the request to the destination - error details |HTTPSConnectionPool(host='ibmx3530m3-52-20', port=5460): Max retries exceeded with url: /ingress/messages (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff276fe33c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))||#012Traceback (most recent call last):#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 396, in plugin_send#012 omf_north.create_omf_objects(raw_data, config_category_name, type_id)#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 221, in wrapper#012 result = _function(*arg)#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 693, in create_omf_objects#012 self._create_omf_objects_automatic(item)#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 541, in _create_omf_objects_automatic#012 typename, omf_type = self._create_omf_type_automatic(asset_info)#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 581, in _create_omf_type_automatic#012 self.send_in_memory_data_to_picromf("Type", omf_type[typename])#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 221, in wrapper#012 result = _function(*arg)#12 File "/usr/local/foglamp/python/foglamp/plugins/north/omf/omf.py", line 754, in send_in_memory_data_to_picromf#012 raise _error#012Exception: an error occurred during the request to the destination - error details |HTTPSConnectionPool(host='ibmx3530m3-52-20', port=5460): Max retries exceeded with url: /ingress/messages (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff276fe33c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))|
Jul 16 13:11:52 scspa0501709001 FogLAMP[31799] ERROR: sending_process: sending_process_1: cannot complete the sending operation of a block of data.
Jul 16 13:11:52 scspa0501709001 FogLAMP[31799] ERROR: sending_process: sending_process_1: cannot complete the sending operation - error details |an error occurred during the request to the destination - error details |HTTPSConnectionPool(host='ibmx3530m3-52-20', port=5460): Max retries exceeded with url: /ingress/messages (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff276fe33c8>: Failed to establish a new connection: [Errno -2] Name or service not known',))||

I see in the http://foglamp.readthedocs.io/en/master/05_testing.html#setting-the-omf-translator-plugin documentation that the example Connector Relay endpoint is HTTP, but I'd like to test against an HTTPS endpoint (using the new version of the Connector Relay that natively supports HTTPS). That endpoint, though, uses a self-signed certificate, so I imagine that certificate validation would fail. When I try to run the test outlined in the documentation, using the new HTTPS endpoint URL for my PI Connector Relay, I don't see any new AF Elements being created. Is there a place where I can check for any OMF-related errors?

In the docs, upon reaching the step where you execute psql to run the commands found in cmds.sql:

http://foglamp.readthedocs.io/en/master/06_plugins.html?highlight=database#building-foglamp-and-adding-the-plugin

I encounter an error

psql: FATAL: database "foglamp" does not exist

Running psql -l, I see that there isn't a foglamp database; are you using the default database instead of one named "foglamp"?

Despite clearing out the field "Static Data" when setting up North Instances, the default data values that were previously there still get sent to the North destination. I'm using an OMF application in the OSIsoft PI Data Collection Manager.

Is it possible to take image file from a central computer to foglamp node. Which plugin is required for that?

When a reading violates the latency limit, a warning message is posted to the log. This message states the latency and the desired latency. Actual latency is in seconds, and desired is in milliseconds, but the message compares them and is stated in seconds.

Example:

Oct 21 10:42:03 XXXX[1145]: WARNING: Current send latency of XXXXX seconds exceeds requested maximum latency of 5000 seconds

Hi,
I have installed FogLAMP from source code. When I run following command it says 404:
curl -s http://localhost:8081/foglamp/category ; echo

curl -s http://localhost:8081/foglamp/schedule | jq
does not list OMF to OCS North Plugin.

What could be the problem. Do I have to enable something special.

Regards,
Priya

I do not know how the .loads works but I suspect it loads the array in memory, which should be avoided.

By removing the line below (pprint), there is a performance boost.

I have setup FogLAMP and also the URL and producerToken to send the data.
There were some problems initially with respect to the security settings of the Pi Points and I set correct Security Settings. Using Data Collection Manager I have created the Pipeline.
However, I might be missing something on the AF setup because whenever I ingest data into FogLAMP and it goes through the north Plugin (OMF) - on FogLAMP the SENT stats increase, but on the OSISoft (DCM) I am seeing this error:
Do I need to create some template - what should be the format, etc.?
NOTE: I am only interested in the data getting ingested into Pi Archive from FogLAMP OMF - but because of following errors Pi Points are not getting created.

Error while checking in the pending changes: Cannot complete the operation because the user does not have rights to write to ElementTemplate 'OMF.FogLAMP Connector.0001_D_CAB/A_typename_sensor' with UniqueID 'bd2be362-4922-4e3a-83cc-baa402d43d77'. AF assets will be checked-in when a new value is sent. Full exception: OSIsoft.Tau.AssetCache.AFCheckInException: Error while checking in the pending changes: Cannot complete the operation because the user does not have rights to write to ElementTemplate 'OMF.FogLAMP Connector.0001_D_CAB/A_typename_sensor' with UniqueID 'bd2be362-4922-4e3a-83cc-baa402d43d77'. ---> System.Security.SecurityException: Cannot complete the operation because the user does not have rights to write to ElementTemplate 'OMF.FogLAMP Connector.0001_D_CAB/A_typename_sensor' with UniqueID 'bd2be362-4922-4e3a-83cc-baa402d43d77'. at OSIsoft.AF.PISystem.CheckServerError(dcServerError err, Boolean firstAttempt) at OSIsoft.AF.Support.AFSerialProxy.Call(String rpcName, ProxyDelegate codeBlock) at OSIsoft.AF.Support.AFSerialProxy.CheckInElementTemplate(dcElementTemplate[] changes, dcObjectIdentity[] deletedIdentities, Guid[] elementsWithAutoCreatedAnalyses, Boolean& updatedUOM, Int64[]& securityIds) at OSIsoft.AF.Asset.AFElementTemplate.RemoteCheckInElementTemplate(AFDatabase database, HashSet1 processedItems, dcElementTemplate[] changes, dcObjectIdentity[] deletedIdentities, Guid[] elementsWithAutoCreatedAnalyses) at OSIsoft.AF.AFDatabase.SaveElementTemplates(Boolean doCheckIn, HashSet1 processedItems, List1 appliedList, HashSet1 elementsWithAutoCreatedAnalysesList, List1& changeList, List1& deleteList) at OSIsoft.AF.AFDatabase.ProcessDatabaseApplyCheckInChanges(ProcessMode mode, AFCheckedOutMode checkedOutMode, IEnumerable1 objList, List1 appliedList, HashSet1 elementsWithAutoCreatedObjects, Boolean& dbPagedRefreshRequired) at OSIsoft.AF.AFDatabase.ProcessDatabaseChanges(ProcessMode mode, AFCheckedOutMode checkedOutMode, IEnumerable1 objList, List1 appliedList, HashSet1 elementsWithAutoCreatedObjects) at OSIsoft.AF.AFDatabase.ProcessDatabaseChanges(ProcessMode mode, AFCheckedOutMode checkedOutMode) at OSIsoft.AF.AFDatabase.CheckIn(AFCheckedOutMode mode) at OSIsoft.Tau.AF.Common.ConnectorDestinationMapper.CommitChanges(AFCheckedOutMode mode) at OSIsoft.Tau.AF.Common.AFContext.CommitChanges(AFCheckedOutMode mode) --- End of inner exception stack trace --- at OSIsoft.Tau.AF.Common.AFContext.CommitChanges(AFCheckedOutMode mode) at OSIsoft.Tau.PI.AssetBuilder.AFAssetBuilder.AddValueLinksToTemplate(AFElementTemplate template, Dictionary2 valueStreamLinks, String connectorType, String connectorUid) at OSIsoft.Tau.PI.AssetBuilder.AFAssetBuilder.AddStreams(AFElement elementToUpdate, Dictionary2 valueStreamLinks, String connectorType, String connectorUid) at OSIsoft.Tau.PI.AssetBuilder.AFAssetBuilder.AddStreams(AssetNode assetNode, Dictionary`2 valueStreamLinks) at OSIsoft.Tau.AssetCache.AssetCache.ProcessLink(AssetLink link) at OSIsoft.Tau.AssetCache.AssetCache.PostLink(TauEvent link, String streamId) at OSIsoft.Tau.AssetCache.AssetCache.FlushAsync(AssetCacheBatch batch)

One more error seen is:
Unable to persist entities to repository: C:\ProgramData\OSIsoft\Tau\Relay.ConnectorHost\SchemaRepository\FogLAMP/schemas.json. Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute. at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource) at System.Collections.Generic.Dictionary2.Enumerator.MoveNext() at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType) at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonConverter[] converters) at OSIsoft.Tau.Relay.Translator.REST.TypeRepository.<>c__DisplayClass26_01.b__0().

@MarkRiddoch, @praveen-garg and @billehunt
If you pre-calculate this line, you have a performance boost of about 15%

@billehunt, @MarkRiddoch and @Terris-Dianomic
Simply comment this line and the print for the PUT method

Hello there

I just wanted to say that there is a dead link on this page which should somehow link here, but content is missing. However I think i found the related content in the github repo under this link, but i might be mistaken.

Im still evaluating the project and thanks a lot for this amazing project.

By the way, is the feature to set Applications via the REST API on any north/south plugins already implemented, and when yes how are the curl commands to set them? An example on maybe how to set a simple Application on the randomwalk or sinusoid south plugin would be highly appreciated.

I can already set it up using this curl command as explained here:

curl -sX POST http://localhost:9081/foglamp/service -d '{"name": "sinusoid", "type": "south", "plugin": "sinusoid", "enabled": true}'

Kind regards

Hi All,

I am posting it here as I couldn't find any community to post clarification. Please let me know if we have one.

Going by the architecture document it sounds to be a very promising tool. But there are some ambiguities couldn't be clarified either from docs or by skimming over the code.

1. Does a south plugin mandatorily formats the sensor reading to a JSON format having keys as asset, timestamp, key & readings ? As it is done in case of DHT11i code.

2. Can't a south plugin pass on the data directly to any of the north plugin without involving storage?
   I couldn't see any south/North plugin config which can configure this behavior. Please enligjhten me if I am missing anything evident.

3. As per architecture document we can plug additional ML software any any third party application using Application module. But there is no description or code explaining how to do it. Is there any API exposed like we have for south and north plugin ?

Hello there

First of all, a very cool project you have. I am trying to evaluate it for an IoT project in school.

My problem so far, I am trying to enable a north plugin (http_north) with the REST API and I am not finding a way of doing it in the documentation nor somewhere else.

I know south plugins are far more easy to setup with the following command:

#Adding South plugin randomwalk from scratch
curl -sX POST http://localhost:8081/foglamp/service -d '{"name": "randomwalk", "type": "south", "plugin": "randomwalk", "enabled": true}'

I do have the plugin installed so far

But how do I accomplish setting up the north_plugin with the REST API? I dont see a POST task or POST schedule, only PUT to update an existing entry in the schedule or task section.

Is this feature missing or am i just on a wrong track?

My best regards

Category output

curl -s http://localhost:8081/foglamp/category | jq
{
  "categories": [
    {
      "key": "SCHEDULER",
      "displayName": "Scheduler",
      "description": "Scheduler configuration"
    },
    {
      "key": "SMNTR",
      "displayName": "Service Monitor",
      "description": "Service Monitor"
    },
    {
      "key": "rest_api",
      "displayName": "Admin API",
      "description": "FogLAMP Admin and User REST API"
    },
    {
      "key": "service",
      "displayName": "FogLAMP Service",
      "description": "FogLAMP Service"
    },
    {
      "key": "General",
      "displayName": "General",
      "description": "General"
    },
    {
      "key": "Advanced",
      "displayName": "Advanced",
      "description": "Advanced"
    },
    {
      "key": "Utilities",
      "displayName": "Utilities",
      "description": "Utilities"
    },
    {
      "key": "randomwalk",
      "displayName": "randomwalk",
      "description": "Generate random walk data points"
    },
    {
      "key": "South",
      "displayName": "South",
      "description": "South microservices"
    },
    {
      "key": "randomwalkAdvanced",
      "displayName": "randomwalkAdvanced",
      "description": "randomwalk advanced config params"
    }
  ]
}

Schedule output

curl -X GET http://localhost:8081/foglamp/schedule | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1394  100  1394    0     0   136k      0 --:--:-- --:--:-- --:--:--  136k
{
  "schedules": [
    {
      "time": 0,
      "exclusive": true,
      "repeat": 3600,
      "id": "cea17db8-6ccc-11e7-907b-a6006ad3dba0",
      "processName": "purge",
      "enabled": true,
      "type": "INTERVAL",
      "day": null,
      "name": "purge"
    },
    {
      "time": 0,
      "exclusive": true,
      "repeat": 15,
      "id": "2176eb68-7303-11e7-8cf7-a6006ad3dba0",
      "processName": "stats collector",
      "enabled": true,
      "type": "INTERVAL",
      "day": null,
      "name": "stats collection"
    },
    {
      "time": 300,
      "exclusive": true,
      "repeat": 43200,
      "id": "2176eb68-7303-11e7-8cf7-a6107ad3db21",
      "processName": "certificate checker",
      "enabled": true,
      "type": "TIMED",
      "day": null,
      "name": "certificate checker"
    },
    {
      "time": 0,
      "exclusive": true,
      "repeat": 0,
      "id": "ea14ad31-4e46-465f-8f8b-98326af74c96",
      "processName": "south_c",
      "enabled": true,
      "type": "STARTUP",
      "day": null,
      "name": "randomwalk"
    },
    {
      "time": 0,
      "exclusive": true,
      "repeat": 3600,
      "id": "d1631422-9ec6-11e7-abc4-cec278b6b50a",
      "processName": "backup",
      "enabled": false,
      "type": "INTERVAL",
      "day": null,
      "name": "backup hourly"
    },
    {
      "time": 0,
      "exclusive": true,
      "repeat": 0,
      "id": "8d4d3ca0-de80-11e7-80c1-9a214cf093ae",
      "processName": "restore",
      "enabled": true,
      "type": "MANUAL",
      "day": null,
      "name": "restore on demand"
    },
    {
      "time": 0,
      "exclusive": true,
      "repeat": 0,
      "id": "fac8dae6-d8d1-11e7-9296-cec278b6b50a",
      "processName": "backup",
      "enabled": true,
      "type": "MANUAL",
      "day": null,
      "name": "backup on demand"
    }
  ]
}