• enumerate the set features that require testing
• implement the test functions

Since sleepnet requires a proxy for time-in-bed as an input, we will need to use the random forest model and its output:

We will take this RF to train a sleep vs non-sleep detector:
https://github.com/OxWearables/stepcount/blob/e53ec0594a2bad40214a7efdda8f7b773869ceb4/stepcount/models.py#L229

• Basic SSl
• SSL+HMM

I believe it is simply a matter of better error handling.

This bug can be reproduced e.g. with the file "ukb/accelerometer/group1/4380443_90001_0_0.cwa.gz"

See attached file for the log.
index_error_log.txt

Please see: OxWearables/stepcount#33

This bug can be reproduced using e.g. the file "ukb/accelerometer/group10/4877373_90001_0_0.cwa.gz"

See attached file for the log.
index_error_bug.txt

This is very interested work, and we are interested in using the algorithm with some of our actigraphy data but have been running into an error. We are using .gt3x files and when running the code, the initial steps seem to work and a handful of outputs are produced: a raw csv, an info JSON file and 2 NPY files (data2model and times), but we then get the error: RuntimeError: torch.cat(): expected a non-empty list of Tensors.

Are you able to advise what the issue may be?

In contrast to other OxWearables packages, asleep produces an "info.json" file for actipy parameters and a "summary.json" file for the sleep-related parameters. I believe these two should be one file.

Also, the --remove_intermediate_files argument deletes "info.json", which I believe is wrong. It is not an intermediate file but an important part of the final output.

I've encountered several critical issues while attempting to use this project, which have significantly hindered my ability to effectively leverage its capabilities. These issues pertain to the lack of dependency pinning, compatibility problems with newer versions of Python, and the overall usefulness of the results produced. Below is a detailed account of each problem and its impact on my experience.

1. Missing Dependency Pinning

It took a while to finde the working dependencies. env.yml.txt

2. Incompatibility with Newer Python Versions

New versions just break it enirely.

3. Missing sample data

The project is missing sample data to verify that it works as expected.

4. Unhelpful Results

Testing muliple datasets the results were always 0 or a few minutes of sleep. As I cannot upload our samples the same happens with sample data from another issue:
#33 (comment)
Output:

Saving files to dir: outputs/sample
Raw data file already exists. Skipping raw data parsing.
   Unnamed: 0                           time         x         y         z  non_wear
0           0  2023-08-15 15:00:00.017000000 -0.461000  0.894996 -0.116996     False
1           1  2023-08-15 15:00:00.050333333 -0.443447  0.781798 -0.121022     False
2           2  2023-08-15 15:00:00.083666666 -0.527073  0.767216 -0.148742     False
3           3  2023-08-15 15:00:00.117000000 -0.605458  0.903918 -0.139534     False
4           4  2023-08-15 15:00:00.150333333 -0.689671  1.284508 -0.111031     False
Data2model file already exists. Skip data transformation.
data2model shape: (8640, 3, 900)
times shape: (8640,)
Non_wear flag shape: (8640,)
prediction set sample count: 8640
Using local /home/tobi/asleep/src/asleep/torch_hub_cache/OxWearables_ssl-wearables_v1.0.0
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [01:38<00:00, 19.62s/it]
(8640,)
(array([0., 1., 2.]), array([5807,  451, 2382]))
                     time  label  is_wear
0 2023-08-15 15:00:00.017      0     True
1 2023-08-15 15:00:30.017      0     True
2 2023-08-15 15:01:00.017      0     True
3 2023-08-15 15:01:30.017      0     True
4 2023-08-15 15:02:00.017      0     True
No sleep windows >30 mins detected. Exiting...
Non-wear time has been written to outputs/sample/non_wear.npy
Current sleep classification has been written to outputs/sample/ssl_sleep.npy

Any thoughts @angerhang

• Describe the set of sleep parameters that would be generated
• Key parameters to use the package

Hang if you're using actipy I don't think Python 3.6 is supported.

Also the java check should be java -version

The asleep package should behave the same way as the step count page at least in terms of input and output interface

• Decide and implement the input output interfaces
• Provide an interface with random forest and sleepnet

Add versioneer to support for automatic publishing for version number increment

Probably a corner case.

This bug can be reproduced e.g. with the file "ukb/accelerometer/group3/5563512_90001_0_0.cwa.gz"

See attached file for the log.
pandas_error.txt

Very interesting package, which I would like to use on data we collected with the Axivity devices. Do you have tips for parallelisation of the get_sleep command to speed up the analysis process of multiple files? I tried to use the command in a shell script using xargs for parallelisation but ran into an error thrown by the Java Runtime (see attached file).

hs_err_pid6048.log

Hey @angerhang , the last bug fix worked well. I have now got through the processing on all our files and have come across a couple of additional issues.

For a number of files we are getting the type error in the title - I have attached a txt output of this. This happens in the JR001.gt3x file that you had previously.

I am also seeing a less frequent issue where I run on a given file and it appears to start, produces the text Saving files to dir: outputs/'filename', but then stops processing and waits for the next file input. It appears that it does not successfully read the file. I didn't know if you may have come across this/had any immediate thoughts? If not I would plan to analyse all the data we can, if we can fix the type error issue and then if all is looking good with those work through everything we need to, to get some of our study files to you to see if you can replicate.

Inconsistent error.txt

I believe this is also a matter of better error handling.

This bug can be reproduced e.g. with the file "ukb/accelerometer/group5/3801805_90001_0_0.cwa.gz"

See attached file for the log.
index_error_log.txt

Before we publish this package we should do the following:

• Make the printout messages more helpful
• Document how to use this package
• Include the entry point get_sleep and visu_sleep
• Fix pypi image

The monitoring StartTime and EndTime is automatically extracted for CWA files but not csv files, leading to a processing error.

Therefore, for csv files, we need to be able to extract this information from the provided 'time' column.

• Calculate the sleep window, sleep window start, sleep window end, and non-wear time per day and the window midpoint
• Test it on the participants that Harry shared so that we make sure that package will run for cases where when sleep end points are easier than the sleep start timestamps.

Run the asleep package on some of the UKB files to see if the results line up

Update the model weights with the latest one.

Currently, the summary statistics only generate the sleep parameters for overnight sleep duration but not the for the total sleep time.

To do this, we will need to add the computation for all the sleep blocks for each day in the day class object.

Right now, get_sleep function is a bit messy.

We probably should clean it up soon/

Having obtained all the required sleep predictions, we will want to generate the common sleep parameters for the downstream analysis

• Common sleep parameters included in the sleepnet paper: tst, se, ratio
• Total sleep time: hourly , daily, weekday, weekend, overall averages
• Check for NAs: at the moment all the NAs are non-wear from actipy. We would want to store the non-wear information in certain format that would allow to do QAs based on the non-wear threshold.
• Test on UKB file

When dealing with NAs, it is probably to specify the the min_duration is 22 hours because it could be that at the start and the end of particular recording, we have a very short day hence also a short NA duration. This might change a bit of the statistics at the beginning and the end.

Having parsed the data into the right format, we will need to

• Run the RF model to obtain sleep window
• Run the sleepnet on continuous blocks of sleep periods

Actipy now renders all the non-wear as NA but this is not ideal because it is likely to yield false positives during sleep periods.

• Disable non-wear detection from parsing using Actipy
• Replicate non-wear detection in asleep
• Correct false positive non-wear detection after the capture-24 model has been applied to account for non-wear periods within two blocks of sleep windows

• Noon2noon interval with SE on the longest sleep episode, time in bed
• Key information to plot: average ENMO, wake/sleep timing (noon2noon); dates and SE.

Is there any chance you could add time zone correction argument(s) similar to GGIR/Accelerometer? Some of our watches were improperly configured, so their time zones need adjusted.

Thanks!