Problem

The search form on the nav bar cannot handle whitespaces appropriately. If you input a query with whitespace(s), such as:

Then you will get no results with + encoded query:

This curious behavior sometimes confuse users who hope the search form to eliminate unnecessary whitespaces automatically. In addition it replaces whitespaces with + curiously.

This application requires to setup some middlewares (MySQL, RabbitMQ, Elasticsearch and Memcached). There is no easy way to build and setup those middlewares for this application for now. This makes an obstract for developer to start to develop.

This issue supposes to make a docker-compose file to build those middlewares which are necessary for run AirOne.

There is a problem that the tasks added in the custom view are executed at the same time even if the targets are the same.
When adding a task in a custom view, I am using existing method of create job.

Dependency is confirmed when creating a job.
Since the operation number is different, the dependency confirmation will be skipped.

When offset is used in the GET entry API, the list limit is fixed at 100.
(e.g. offset=90 -> list=90~100)

curl 'http://127.0.0.1:8000/api/v1/entry?entity=TestEntity&offset=90' \
-H 'Authorization: Token xxxxx'

In the case of an entry that has array attributes which contain HTML Special values as below,

These values are encoded when you open the edit page of it like this.

Reproduction procedure

1. Create an Entity.
   Name: TestEntity

2. Create an Attribute.
   Name: ref, Type: entry, refer: TestEntity

3. Create an Entry.
   Name: TestEntity

4. Import the YAML file.

EntityAttr:
- create_user: user_name
  id: xxxx
  entity: TestEntity
  is_mandatory: false
  name: ref
  refer: TestEntity
  type: 1

Background

AirOne is currently sending all DB queries to the master database.
As the number of users increases, the performance of the master database becomes a bottleneck.
The select queries should be sent to the slave database for reduce the load on the master database.

However, there is a problem sending all select queries to the slave database.
AirOne's database is MySQL.
The master / slave configuration is thinking of MySQL replication.
MySQL replication can cause replication delays and should be considered.
When databases is delay, A problem occurs a process called SELECT (delay) → UPDATE.

Solution

Only the GET method uses the slave database.
The reason is that the GET method has no update process.
However, AirOne has a process to update with the GET method and needs to be fixed.

Make a configuration to build and deploy document to gh-pages branch to update documentation page When master branch is updated.

LDAP error log is output when authentication failed.
It is necessary to change the log level of authentication failure.

[ERROR] asctime:2021-07-26 10:09:48,521 module:ldap     message:{'msgtype': 97, 'msgid': 1, 'result': 49, 'desc': 'Invalid credentials', 'ctrls': []}   process:16860   thread:139823662491392
[INFO]  asctime:2021-07-26 10:09:48,522 module:ldap     message:Failed to authenticate user(hinagawa-koya) in LDAP      process:16860   thread:139823662491392

Summary

We require to implement pagination feature at the list page of Entries in the specified Entity.

Background

AirOne implements search feature to be able to search and narrow down Entries user wants to find as below.

Problem

The Entry list page has limitation to be able to list Entries at most 50 ones even through more than 50+ Entries which are matched with keyword have been registered. Because of this, that narrowing down implementation is not enough for UX in the following use-cases.

• User wants to know how many entries are registered at AirOne that has "template-" prefixed name.
  • User can only know 50 Entries (other registered entries are invisible and unnoticeable.

Requirements to fix

It's necessary to be able to list Entries beyond the limitation of the number to list Entries at once without spoiling performance.

An idea to solve this problem is pagination, I think.

Make a document that describes how to install AirOne

The current version of django that AirOne uses is v1.11(LTS). But it is obsoleted and also its support has already been ended.
(c.f. https://www.djangoproject.com/download/)

We should upgrade it to next LTS (v2.2.x).

I think it's impossible for user to start to use this software without tutorial. If we want to spread this software among with the people who suffer from the problem of information management, tutorial and introduction page are necessary.

Background / Problem

After adding an EntityAttrs at editing entity, it takes long time to show Entry of edited Entity.
That is because Attributes of entry would be created when user refers it.

Therefore, in the case to show an Entry (or show CustomView Entry that refers entries of edited Entity) takes very long time (may be couple of minutes).

Solution

This problem could be alleviate by following solutions.

• Create a new job that complement Attributes of Entries of edited Entity.
• Call above job after editing an Entity

The method of getattr of Attribute class is a helper method to get latest AttributeValue of specifying Attribute.
It should not return AttributeValue which is related with deleted Attribute. However it happend just after an EntityAttr was deleted and before viewing target entry (before deleting EntityAttr propaget to entry).

Reproduction procedure

1. Create an entity (E1) with EntityAttr(A1).
2. Create an entry (e1) of E1 and set value (v1) to A1.
3. Delete A1 by editing E1.
4. Run following script through django shell

from entry.models import Entry

e = Entry.objects.get(id=2155631)
e.get_attrv('A1').__class__

The last processing try to get value of deleted attribute. So it's expected that 'NoneType' value would be returned. But it returns AttributeValue of v1.

A following exception will be happened at the processing of creating an Entry on a background process.

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/user/airone/job/models.py", line 208, in run
    return method(self.id)
  File "/home/user/airone/.venv/lib/python3.6/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/home/user/airone/.venv/lib/python3.6/site-packages/celery/app/task.py", line 375, in __call__
    return self.run(*args, **kwargs)
  File "/home/user/airone/entry/tasks.py", line 115, in create_entry_attrs
    if not any([int(x['id']) == attr.schema.id for x in recv_data['attrs']]):
  File "/home/user/airone/entry/tasks.py", line 115, in <listcomp>
    if not any([int(x['id']) == attr.schema.id for x in recv_data['attrs']]):
AttributeError: 'NoneType' object has no attribute 'schema'

It could be reproduced when multiple requests that targets to create same Entry were queued to process at background processes.
The background processing of creating Entry should be safe when that situation was happened.

When user registers many Entities at AirOne, showing the list entity page might be slow because it try to show whole entity at once.

Abstract

Once user create an entry via API (post /api/v1/entry), an Attribute value would be invisible when Nothing is set on the general authorization of its EntityAttr. Because there is a bug individual authorization never be inherited.

Reproduction procedure

1. Set Nothing on general authorization setting of a specific Attribute (in this case attr1 of Entity EntityTest1) and also set Full Controllable to the user of logged in.

2. Create an Entry of EntityTest1 through API as below

curl -X POST 'http://localhost:8080/api/v1/entry' \
-H 'Authorization: Token <AccessToken>' \
-H "Content-type: application/json" \
-d '{ "entity": " EntityTest1", "name": "test2", "attrs": {"attr1": "foo"} }'

Then, you could see that created entry doesn't have any attribute.

But, actually, there is an Attribute and value is also set correctly. The reason why user couldn't see it is that individual authentication configuration is not inherited. You can confirm it when you switch login user to administrative one.

Note

This problem happens only when user create an entry via API. I confirmed it doesn't happen through Web UI and also editing via API.

When you specify -1 at the page parameter in the Entity list page, following exception would be happend

Environment:


Request Method: GET
Request URL: http://gs2:21000/entity/?page=-1

Django Version: 1.11.29
Python Version: 3.6.9
Installed Applications:
['common',
 'user',
 'group',
 'entity',
 'acl',
 'dashboard',
 'entry',
 'job',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'import_export',
 'rest_framework',
 'rest_framework.authtoken',
 'custom_view.background',
 'frontend']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/home/dooga/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/home/dooga/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/home/dooga/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/dooga/airone/airone/lib/profile.py" in wrapper
  30.         ret = func(*args, **kwargs)

File "/home/dooga/airone/airone/lib/http.py" in wrapper
  35.         return func(*args, **kwargs)

File "/home/dooga/airone/entity/views.py" in index
  53.     return_entities = overall_entities[index_start:index_start + CONFIG.MAX_LIST_ENTITIES]

File "/home/dooga/airone/virtualenv/lib/python3.6/site-packages/django/db/models/query.py" in __getitem__
  269.             "Negative indexing is not supported."

Exception Type: AssertionError at /entity/
Exception Value: Negative indexing is not supported.

Once unauthorized user try to create an user by submitting button in the page of creating user, there is no response in a browser. It is caused by the JavaScript never catch the response which is replied by server. It should show an Error message once it happened.

Summary

We decide to change specification of listing Attributes and its values of specified Entry about ACL as below.

AsIs

• When accessed user doesn't have permission to show an Attribute, AirOne hide its Attribute name and its AttributeValue.

ToBe

• When accessed user doesn't have permission to show an Attribute, AirOne show it Attribute name but hide its AttributeValue.

Background

Because of the current specification of ACL for listing Attributes and its values, user would confused when they realize some Attributes are vanished. And they might think that it's a bug even through the correct behavior of authorization, or they might create another same named Attribute.
To avoid those situation, we decide to change this spec to show what Attribute is prohibited to show to the users who don't have permission to show by ACL.

The way of reproduction

1. Create an Entity(E1) that has an Attribute(A1) of date type.
2. Create an Entry(e1) and set an arbitrary value (e.g. 2020-01-01) at A1.
3. Request advanced search with E1 and A1.
4. Do CSV export.

Date typed value was not output of exporting CSV file.

Search result of e1(hoge)

CSV export reslut

Name,Entity,date
hoge,TestEntity

summary

We want to replace ldap3 library to following python-ldap library.
https://pypi.org/project/python-ldap/

(c.f. https://github.com/dmm-com/airone/blob/master/requirements.txt#L3)

and keep it on maintaining.

Goal

Being able to confirm test results for each PR

The way of reproduction

1. Create an Entity(E1) that has an Attribute(A1) of date type.
2. Create an Entry(e1) and set an arbitrary value (e.g. 2020-01-01) at A1.
3. Request advanced search with E1 and A1.

Then the value which is set in the procedure (2) is expected to be shown, but it doesn't as below.

Search result of e1(hoge)

Registered information of e1(hoge)

module:urls messages are not so frustration for me.

But module:ldap them are.

example.

$ python manage.py test
Creating test database for alias 'default'...
[INFO]  asctime:2020-02-26 14:08:37,785 module:urls     message:advanced API endpoints are unavailable  process:37344   thread:4502900160
[INFO]  asctime:2020-02-26 14:08:37,793 module:urls     message:There is no URL dispatcher of custom-view       process:37344   thread:4502900160
System check identified no issues (0 silenced).
...........................................[INFO]       asctime:2020-02-26 14:08:44,462 module:ldap     message:Failed to authenticate user(ldap_user) in LDAP  process:37344   thread:4502900160
..[ERROR]       asctime:2020-02-26 14:08:44,676 module:ldap     message:('unable to open socket', [(LDAPSocketOpenError('socket connection error while opening: [Errno 61] Connection refused',), ('::1', 389, 0, 0)), (LDAPSocketOpenError('socket connection error while opening: [Errno 61] Connection refused',), ('127.0.0.1', 389))])     process:37344   thread:4502900160
[INFO]  asctime:2020-02-26 14:08:44,676 module:ldap     message:Failed to authenticate user(invalid_user) in LDAP       process:37344   thread:4502900160
[ERROR] asctime:2020-02-26 14:08:44,679 module:ldap     message:('unable to open socket', [(LDAPSocketOpenError('socket connection error while opening: [Errno 61] Connection refused',), ('::1', 389, 0, 0)), (LDAPSocketOpenError('socket connection error while opening: [Errno 61] Connection refused',), ('127.0.0.1', 389))])     process:37344   thread:4502900160
[INFO]  asctime:2020-02-26 14:08:44,680 module:ldap     message:Failed to authenticate user(guest) in LDAP      process:37344   thread:4502900160
.......................................................................................................................................................................................s...............................................s..............................................................................................................................................
----------------------------------------------------------------------
Ran 419 tests in 233.525s

OK (skipped=2)
Destroying test database for alias 'default'...

There are many module:ldap messages that outputs seem to be not clean.

ref PR. #15

If you are using entry type attributes when editing an entry
If the search target exceeds the upper limit, it may not be displayed in the results.

The upper limit of the search target is set below.

e.g.) 172.27.19.1 is missing from search results

I get an exception error when requesting a GET entry API with an invalid offset value.

e.g.)

http://127.0.0.1:8000/api/v1/entry?entity=TestEntity&offset=-1
http://127.0.0.1:8000/api/v1/entry?entity=TestEntity&offset=test

Only numbers should be allowed.

Filtering attributes on advanced search result page basically performs partial match. For e.g. hoge keyword returns a result has hogefuga. But date-typed attribute has a different behavior.

When I get these entries:

I put 2021-03 keyword with expecting 2 results has 2021-03-27, but actually it returned nothing:

If I put the full date string 2021-03-27, it returns the results.

I thinks its not natural, should follow other kinds of attributes.

Webhooks allow AirOne to send requests to external websites.
External websites may use self-signed certificates and will output SSL warning logs.
Stop SSL warning logging as it interferes with log monitoring.

When user sends a request to an API endpoint, AirOne output following degenerated log that doesn't contain following informations.

• Who sends this request
• Which endpoint is called
• What type of HTTP method

[INFO]  asctime:2021-01-28 12:09:52,452 module:profile  message:(Profiling result: 0.007774s) Total time of the request process:11902   thread:139678475540288

way of reproducing

1. Make an entity (E1) that has attribute typed array_string
2. Make an entry(e1) of E1 and set value (a backslash \) to the attribute.
3. Show edit tab of e1, then attribute value couldn't be shown (because of a JavaScript problem was happend).

(c.f. #13 (comment))

For reactization #133, "get a group" and "list gruops" APIs are useful to reimplement user management pages.

Rough specs are:

/group/api/v2/groups:

[
    {
        id: 1,
        name: 'test',
        members: [
            {
                name: 'user1',
            },
            {
                name: 'user2',
            },
        ],
        ...
    },
    ...
]

/group/api/v2/groups/1:

{
    id: 1,
    name: 'test',
    members: [
        {
            name: 'user1',
        },
        {
            name: 'user2',
        },
    ],
    ...
}

I think it's better to make a test to confirm that an e-mail which has expected context will be dispatched when an error is happened in celery process at #102.

The following document is helpful to make a test to confirm context of sending e-mail.
https://docs.djangoproject.com/en/3.2/topics/testing/tools/#topics-testing-email

Because of the change of #44, a bug that referral Entity of an EntityAttr can't be changed in editing Entity was embedded.

That PR was made to prohibit to change type of EntityAttr to keep consistency of AttributeValues which are related to it.
But referral Entities of EntityAttr should enable to be changed for effective data management.
(e.g. A case to add Entry to refer from a specific Attribute because of changing of management policy)

Exception error is output because there is no user authentication validation in some APIs.
e.g.

curl -X POST 'http://(AirOneURL)/api/v1/entry' \
-H 'Authorization: Token xxxxx' \
-H "Content-type: application/json" \
-d '{ "entity": "TestEntity", "name": "TestEntry", "attrs":{}}'

Enter an incorrect value for Token.

There is a bug that internal error would be happened by following procedures.

1. Create an Entity "E1" with an "array_string" typed attribute that is named by "attr".
2. Create an Entry "e1" and set arbitrary value to attribute "attr".
3. Edit Entity "E1" and delete attribute "attr", then save it.
4. Edit Entity "E1" and add a "string" typed attribute that is named by "attr".
5. Edit Entry "e1" and save it without any input.
6. Send request of advanced search of "E1" with attribute "attr".

After the procedure (6) following exception would be happened.

Internal Server Error: /dashboard/advanced_search_result
Traceback (most recent call last):
  File "/home/user/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/user/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/user/airone/virtualenv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/user/airone/airone/lib/http.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/home/user/airone/airone/lib/profile.py", line 29, in wrapper
    ret = func(*args, **kwargs)
  File "/home/user/airone/dashboard/views.py", line 175, in advanced_search_result
    print('(%s|%s) %s' % (name, val['value'].__class__.__name__, val['value']))
KeyError: 'value'
[ERROR] asctime:2020-10-12 00:24:59,494 module:basehttp message:"GET /dashboard/advanced_search_result?has_referral=false&is_all_entities=false&attr[]=date&attr[]=date2&entity[]=2175735 HTTP/1.1" 500 87837   process:21295   thread:140670217373440

Make a document that describes abstract of AirOne

Some use cases require another data type for Attribute to be able to describe array of group (e.g. "Use-Case: what (user) groups are involved in").

When an Entity has attribute of array_string type as mandatory, an entry of this entity cannot create (or update) even though attribute values are set as below.

All attributes are searched by simple search in the custom view.
Searching for a long keyword string times out the query to elasitcsearch.

hints = [{'name': x.name, 'keyword': query} for x in EntityAttr.objects.filter(is_active=True)]
Entry.search_entries(user, entity_ids, hints, or_match=True)

The AirOne implements a simple profiling feature to log following informations for each requests by using the airone_profile decorator.

• Who authenticated user access
• Which path is accessed
• How many time it take until a response is returned

But its decorator doesn't apply some API handlers (e.g. AccessTokenAPI). We have to coverage whole request handlers.

For reactization #133, "get a user" and "list users" APIs are useful to reimplement user management pages.

Rough specs are:

/user/api/v2/users

[
  {
    id: 42,
    name: 'name',
    email: '[email protected]',
    created_at: '2021-01-02 12:34:56',
    ...
  },
  ...
]

/user/api/v2/users/42

{
  id: 42,
  name: 'name',
  email: '[email protected]',
  created_at: '2021-01-02 12:34:56',
  ...
}

Background / Problem

When user try to register a new user with empty form. Server responds with HTTP 400 error. But no error message will be revealed because there is no error handling implementation in the processing of JavaScript.

Requirements of fix

At least, AirOne should implement an error handler to be able to let user to know followings.

• What problem was happened.
• What should user do.

Currently user can change EntityAttr type by editing Entity page. And once type of EntityAttr was changed, the values that have been registered at corresponding Attribute would be invalid.

This behavior is as specified. But changing Attribute type is not supposed to do after registering Attribute value. Therefore, this operation should be prohibited by design.

The implementation of Entry.add_value method is too long. This method do following processing.

1. Update flags of current latest AttributeValue.
2. Convert input value in response to the attribute-type.
3. Create new AttributeValue instance and register value of (2).
4. Register new AttributeValue to Attribute as the latest one.

In the current implementation, these proceedings were written without separation. Therefore, this method became too long.

This should be refactored by separating each functions to increase maintainability.