Giter Club home page Giter Club logo

course-discovery's Introduction

Course Discovery Service

Service providing access to consolidated course and program metadata.

Getting Started with Development

This repository works with openedx devstack. Once the devstack has been set up and provisioned, run the following commands in devstack directory to access Discovery shell and perform operations as needed

$ make dev.up.discovery
$ make discovery-shell
$ make requirements
$ make validate

Using elasticsearch locally

To use elasticsearch locally, and to update your index after adding new data that you want elasticsearch to access run:

$ ./manage.py update_index --disable-change-limit

To delete elasticsearch old indexes locally you have to use

$ ./manage.py remove_unused_indexes

This command will purge the oldest indexes, freeing up disk space. This command will never delete the currently used indexes.

Also you can use base commands by django-elasticsearch-dsl library.

Delete all the currently used indexes in Elasticsearch:

$ ./manage.py search_index --delete [-f] [--models [app[.model] app[.model] ...]]

Create the indices and their mapping in Elasticsearch:

$ ./manage.py search_index --create [--models [app[.model] app[.model] ...]]

Populate the Elasticsearch mappings with the django models data (index need to be existing):

$ ./manage.py search_index --populate [--models [app[.model] app[.model] ...]] [--parallel]

Recreate and repopulate the indices:

$ ./manage.py search_index --rebuild [-f] [--models [app[.model] app[.model] ...]] [--parallel]

Please use the link to get more https://django-elasticsearch-dsl.readthedocs.io/en/latest/management.html

WARNING: Be aware that search_index command works without sanity index check. So be careful to use it.

Working with memcached locally

Some endpoints, such as /api/v1/courses, have their responses cached in memcached through mechanisms such as the CompressedCacheResponseMixin. This caching may make it difficult to see code changes reflected in various endpoints without first clearing the cache or updating the cache keys. You can update the cache keys by going to any course_metadata model in the admin dashboard and clicking save. To flush your local memcached, make sure the edx.devstack.memcached container is up and run:

$ telnet localhost 11211
$ flush_all
$ quit

Running Tests Locally, Fast

There is a test settings file course_discovery.settings.test_local that allows you to persist the test database between runs of the unittests (as long as you don't restart your container). It stores the SQLite database file at /dev/shm, which is a filesystem backed by RAM. Using this test file in conjunction with pytest's --reuse-db option can significantly cut down on local testing iteration time. You can use this as follows: pytest course_discovery/apps/course_metadata/tests/test_utils.py --ds=course_discovery.settings.test_local --reuse-db

The first run will incur the normal cost of database creation (typically around 30 seconds), but the second run will completely skip that startup cost, since the --reuse-db option causes pytest to use the already persisted database in the /dev/shm directory. If you need to change models or create databases between runs, you can tell pytest to recreate the database with -recreate-db.

Debugging Tests Locally

Pytest in this repository uses the pytest-xdist package for distributed testing. This is configured in the pytest.ini file. However, pytest-xdist does not support pdb.set_trace(). In order to use pdb when debugging Python unit tests, you can use the pytest-no-xdist.ini file instead. Use the -c option to the pytest command to specify which ini file to use.

For example,

pytest -c pytest-no-xdist.ini --ds=course_discovery.settings.test --durations=25 course_discovery/apps/publisher/tests/test_views.py::CourseRunDetailTests::test_detail_page_with_comments

Getting Help

Documentation is hosted on Read the Docs. The source is hosted in this repo's docs directory. The docs are automatically rebuilt and redeployed when commits are merged to master. To contribute, please open a PR against this repo.

License

The code in this repository is licensed under version 3 of the AGPL unless otherwise noted. Please see the LICENSE file for details.

Contributing

Contributions are very welcome. Please read How To Contribute for details.

This project is currently accepting all types of contributions, bug fixes, security fixes, maintenance work, or new features. However, please make sure to have a discussion about your new feature idea with the maintainers prior to beginning development to maximize the chances of your change being accepted. You can start a conversation by creating a new issue on this repo summarizing your idea.

The Open edX Code of Conduct

All community members are expected to follow the Open edX Code of Conduct.

Reporting Security Issues

Please do not report security issues in public. Please email [email protected].

More Help

If you're having trouble, we have discussion forums at discuss.openedx.org where you can connect with others in the community.

Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.

For anything non-trivial, the best path is to open an issue in this repository with as many details about the issue you are facing as you can provide.

For more information about these options, see the Getting Help page.

course-discovery's People

Contributors

afaqshuaib09 avatar ali-d-akbar avatar aliadnansohail avatar asadazam93 avatar attiyaishaque avatar awais786 avatar azanbinzahid avatar bbaker6225 avatar bderusha avatar clintonb avatar cpennington avatar dawoudsheraz avatar dianakhuang avatar dillon-dumesnil avatar edx-requirements-bot avatar edx-semantic-release avatar hammadahmadwaqas avatar matthewpiatetsky avatar mikedikan avatar mikix avatar muhammad-ammar avatar renovate-bot avatar renovate[bot] avatar rgraber avatar saleem-latif avatar schenedx avatar uzairr avatar waheedahmed avatar xnick421 avatar zawan-ila avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

course-discovery's Issues

renovate PRs automatically rebase and kick off CI every time `master` is updated

As of writing, there are 10 open renovate PRs.

When anything is merged to master, the renovate bot rebases the PRs which triggers the CI.

This is problematic for a few reasons:

  • With 10 open renovate PRs, a large number of jobs are kicked off simultaneously every time something is merged to master.
  • Renovate PRs that are open for a long time are responsible for a ton of CI runs

This would not be an issue if renovate PRs were merged in a timely fashion.

TransportError(404, 'IndexMissingException[[*] missing]') running make migrate

I'm using edxapp open-release/ficus.master on ubuntu 16.04. When doing make migrate on discovery version [master, 2017-03-03, release-2017-02-28..] the migration passes fine but i'm getting

elasticsearch.exceptions.NotFoundError: TransportError(404, 'IndexMissingException[[*] missing]')

while doing

python manage.py install_es_indexes'

Output below

python manage.py migrate --noinput

(....)
2017-04-26 18:42:47,134 INFO 15186 [course_discovery.apps.core.management.commands.install_es_indexes] /edx/app/discovery/discovery/course_discovery/apps/core/management/commands/install_es_indexes.py:19 - Attempting to establish initial connection to Elasticsearch host [http://127.0.0.1:9200/]... 2017-04-26 18:42:47,134 INFO 15186 [course_discovery.apps.core.management.commands.install_es_indexes] /edx/app/discovery/discovery/course_discovery/apps/core/management/commands/install_es_indexes.py:21 - ...success! 2017-04-26 18:42:47,134 INFO 15186 [course_discovery.apps.core.utils] /edx/app/discovery/discovery/course_discovery/apps/core/utils.py:12 - Making sure alias [catalog] exists... 2017-04-26 18:42:47,138 WARNING 15186 [elasticsearch] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/base.py:82 - HEAD /_alias/catalog [status:404 request:0.003s] 2017-04-26 18:42:47,138 WARNING 15186 [elasticsearch] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/base.py:82 - HEAD /_alias/catalog [status:404 request:0.003s] 2017-04-26 18:42:47,188 INFO 15186 [course_discovery.apps.core.utils] /edx/app/discovery/discovery/course_discovery/apps/core/utils.py:23 - ...index [catalog_20170426184247] created. 2017-04-26 18:42:47,192 WARNING 15186 [elasticsearch] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/base.py:82 - POST /_aliases [status:404 request:0.004s] 2017-04-26 18:42:47,192 WARNING 15186 [elasticsearch] /edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/base.py:82 - POST /_aliases [status:404 request:0.004s] Traceback (most recent call last): File "manage.py", line 15, in <module> execute_from_command_line(sys.argv) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line utility.execute() File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv self.execute(*args, **cmd_options) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute output = self.handle(*args, **options) File "/edx/app/discovery/discovery/course_discovery/apps/core/management/commands/install_es_indexes.py", line 23, in handle ElasticsearchUtils.create_alias_and_index(es, alias) File "/edx/app/discovery/discovery/course_discovery/apps/core/utils.py", line 32, in create_alias_and_index es.indices.update_aliases(body) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped return func(*args, params=params, **kwargs) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/client/indices.py", line 465, in update_aliases params=params, body=body) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/transport.py", line 307, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request self._raise_error(response.status, raw_data) File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.NotFoundError: TransportError(404, 'IndexMissingException[[*] missing]') Makefile:70: recipe for target 'migrate' failed make: *** [migrate] Error 1

This error was mentioned before on the slack channel but still no answer for it, just a hack

create_or_update_partner error when 'code' is too long

The create_or_update_partner command doesn't gracefully handle a 'code' string that it considers too long. The first time I ran this command (i.e. no existing partners), I used a code 15 characters long. Here's the error I got...

Traceback (most recent call last):
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 480, in update_or_create
    obj = self.get(**lookup)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 387, in get
    self.model._meta.object_name
course_discovery.apps.core.models.DoesNotExist: Partner matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/edx/app/discovery/discovery/course_discovery/apps/core/management/commands/create_or_update_partner.py", line 113, in handle
    'oidc_secret': options.get('oidc_secret'),
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 482, in update_or_create
    obj, created = self._create_object_from_params(lookup, params)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 499, in _create_object_from_params
    obj = self.create(**params)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 401, in create
    obj.save(force_insert=True, using=self.db)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django_extensions/db/models.py", line 24, in save
    super(TimeStampedModel, self).save(**kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/base.py", line 708, in save
    force_update=force_update, update_fields=update_fields)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/base.py", line 736, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/base.py", line 820, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/base.py", line 859, in _do_insert
    using=using, raw=raw)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/query.py", line 1039, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1060, in execute_sql
    cursor.execute(sql, params)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 112, in execute
    return self.cursor.execute(query, args)
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/MySQLdb/cursors.py", line 228, in execute
    if not self._defer_warnings: self._warning_check()
  File "/edx/app/discovery/venvs/discovery/lib/python3.5/site-packages/MySQLdb/cursors.py", line 127, in _warning_check
    warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Data truncated for column 'short_code' at row 1

Make migrate fails at first migration

When I try to run make migrate I get this error on a fresh install on latest master:

Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial...Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/migrations/executor.py", line 154, in apply_migration
    self.recorder.record_applied(migration.app_label, migration.name)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/migrations/recorder.py", line 67, in record_applied
    self.migration_qs.create(app=app, name=name)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/models/query.py", line 348, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/db/models/base.py", line 771, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/haystack/signals.py", line 51, in handle_save
    index = self.connections[using].get_unified_index().get_index(sender)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/haystack/utils/loading.py", line 112, in __getitem__
    self._connections[key] = load_backend(self.connections_info[key]['ENGINE'])(using=key)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/haystack/utils/loading.py", line 65, in load_backend
    return import_class(full_backend_path)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/site-packages/haystack/utils/loading.py", line 32, in import_class
    module_itself = importlib.import_module(module_path)
  File "/home/vedran/Develop/course-discovery/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'course_discovery.apps.core.haystack_backends'
Makefile:58: recipe for target 'migrate' failed
make: *** [migrate] Error 1

This is the migration that is failing: https://github.com/django/django/blob/master/django/contrib/contenttypes/migrations/0001_initial.py
I don't know why a Django migration should fail here?

docker-compose version incompatible with docker-compose.yml

docker-compose <1.6.0 gets installed, but the docker-compose.yml uses version 2 of the configuration format which requires docker-compose >= 1.6.0.

docker-compose version which gets installed: https://github.com/edx/course-discovery/blob/3770f725/requirements/local.txt#L12
configuration version: https://github.com/edx/course-discovery/blob/3770f725/docker-compose.yml#L1
compatibility documentation: https://docs.docker.com/compose/compose-file/#version-2

I upgraded to docker-compose 1.6.0 and it seems to work (before it complained that "version" is not a service, or something).

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

docker-compose
.ci/docker-compose-ci.yml
  • mysql 8.0
  • elasticsearch 7.17.22
  • memcached 1.6
docker-compose.yml
  • mysql 5.7
  • elasticsearch 7.17.22
  • memcached 1.6.28
dockerfile
Dockerfile
  • ubuntu focal
github-actions
.github/workflows/add-depr-ticket-to-depr-board.yml
.github/workflows/add-remove-label-on-comment.yml
.github/workflows/ci.yml
  • actions/checkout v3
  • actions/upload-artifact v2
  • actions/checkout v3
  • actions/download-artifact v2
  • codecov/codecov-action v1
  • actions/checkout v3
  • actions/checkout v3
.github/workflows/commitlint.yml
.github/workflows/docker-publish.yml
  • actions/checkout v3
  • actions/github-script v6
  • docker/build-push-action v1
.github/workflows/migrations-check-mysql8.yml
  • actions/checkout v3
  • actions/setup-python v2
  • actions/cache v2
.github/workflows/requirements-upgrade.yml
  • actions/checkout v3
  • actions/setup-python v2
  • dawidd6/action-send-mail v3
  • ubuntu 20.04
.github/workflows/self-assign-issue.yml
npm
package.json
  • ansi-regex 6.0.1
  • bootstrap-sass 3.4.3
  • bower 1.8.12
  • css-loader 6.11.0
  • datatables.net 1.13.11
  • datatables.net-bs 1.13.11
  • mini-css-extract-plugin 2.9.0
  • file-loader 6.2.0
  • imports-loader 0.8.0
  • jquery 3.7.1
  • sass 1.77.6
  • sass-loader 12.1.0
  • url-loader 4.1.1
  • webpack 5.53.0
  • webpack-bundle-tracker 1.8.1
  • webpack-cli 4.8.0
  • geckodriver 3.2.0
pip_requirements
requirements.txt
requirements/monitoring/requirements.txt

  • Check this box to trigger a request for Renovate to run again on this repository

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.