cxuesong / wikiclientlibrary Goto Github PK

View Code? Open in Web Editor NEW

78.0 7.0 15.0 61.98 MB

/*🌻*/ Wiki Client Library is a portable & asynchronous MediaWiki API client library on .NET Standard.

Home Page: https://github.com/CXuesong/WikiClientLibrary/wiki

License: Apache License 2.0

C# 99.21% PowerShell 0.79%

mediawiki-api mediawiki mediawiki-bot wikibase wikidata-bot wikidata wikia fandom-wiki scribunto structured-discussion

wikiclientlibrary's Introduction

Wiki Client Library

A hand-crafted asynchronous MediaWiki API client library for wiki sites (including Wikipedia and its sister projects, as well as FANDOM and Wikia.org. The library targets at .NET Standard 2.1 & .NET 5.0 (See Supported Platforms), and focuses on the API compatibility with MediaWiki 1.19, as well as the state-of-art APIs of MediaWiki (i.e. 1.34-wmf, as in 2019-05). Other versions in between are hopefully also compatible.

For migrated FANDOM sites on MW 1.33+ (See UCP migration post), you may need to use Bot Password in order to login successfully.

If you are on prior versions of .NET Core or .NET Framework, you may use prior version of the WCL packages. WCL v0.7.x libraries target .NET Standard 1.1 and should support lower versions of the .NET implementations.

The packages CXuesong.MW.WikiClientLibrary.* are now available on NuGet. E.g. you may install the main package using the following command

#  Package Management Console
Install-Package CXuesong.MW.WikiClientLibrary
#  .NET CLI
dotnet add package CXuesong.MW.WikiClientLibrary

Package	Status
CXuesong.MW.WikiClientLibrary
CXuesong.MW.WikiClientLibrary.Cargo
CXuesong.MW.WikiClientLibrary.Flow
CXuesong.MW.WikiClientLibrary.Wikia
CXuesong.MW.WikiClientLibrary.Wikibase

If you bump into bugs, have any suggestions or feature requests, feel free to open an issue. Any contributions on documentations (code annotations & repository wiki) are also welcomed. Thank you.

Overview

Developed in Visual Studio 2019, this portable & asynchronous MediaWiki API client provides an easy and asynchronous access to commonly-used MediaWiki API. The library has the following features

Queries for and edits to pages, categories, and files; page information inspection; file uploading.
Login/logout via simple asynchronous functions.
- Client code has access to CookieContainer, and therefore has chance to persist it.
Tokens are encapsulated in the library functions, so that client won't bother to retrieve them over and over again.
Query continuations are encapsulated by IAsyncEnumerable, which will ease the pain when using page generators.
Other miscellaneous MediaWiki API, such as
- OpenSearch
- Page parsing
- Patrol
Scribunto Lua console and server-side module execution support
StructuredDiscussions (aka. Flow) support
Basic Wikibase (Wikidata's back-end) API support; the library provides facility to consume Wikibase JSON dump
Basic FANDOM / Wikia API (Nirvana, Wikia AJAX, and Wikia REST-ful API v1) support

wikiclientlibrary's People

Contributors

Stargazers

Watchers

Forkers

ntrpnr aquilla12 riesaexe sivukhin jtone123 fafre ostrzyciel user123698745 southsideringo usernamekiran spocweb ashotjanibekyan dgerding ayaz345 hymccord

wikiclientlibrary's Issues

Use more precise data type in WbQuantity

Extracted from #24 & #25.

Use more precise data type in WbQuantity. For now it uses double, but we need something like BigFloat.

But it's a little bit difficult to find a suitable data type to represent floats with arbitrary precision in .NET. Perhaps we need to make one out of BigInteger.

weird file: zh/WikiClientLibrary.resources.dll

When using the latest pre-release version of WCL (v0.7.0-int.6) I'm finding the following file and folder in my bin directory. Is it supposed to be there? Do I need if when publishing my app?
zh/WikiClientLibrary.resources.dll
Probably related to localization resx files...

Issues with wikia.com migration to fandom.com

The new domains seem to be causing issues. The library can still access wikis on *.wikia.com but throws various exceptions when trying to access *.fandom.com. So far I've had an error from Json.NET:

Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.Linq.JToken.ReadFrom(JsonReader reader, JsonLoadSettings se...

when using my own code with a WikiaSite. I got another different error when doing the same thing with a WikiSite in the library's ConsoleTestApplication1:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

But when using a WikiaSite instead the first error could be reproduced.
I don't currently have time to sort this out, but I'm working on debugging when I can.

Consider using Microsoft.Extensions.Logging.ILogger

Instead of defining an interface and implement it, consider using Microsoft's interface thus allowing clients to easily use their logger without an adapter.
Also there's a great implementation for this logger using NLog which is very powerful.

WikiLink should fully support interwiki links?

As a temporary solution, WikiLink class now handles interwiki links as an interwiki prefix and a title, regardless of the namespace name.

E.g. we have [[site:Project:Sandbox]]. In the current commit b1bd854, the title is handled like this

InterwikiPrefix = site
NamespaceName = [null]
Title = Project:Sandbox

But as a perfect solution, we want something like this

InterwikiPrefix = site
NamespaceName = Project
Title = Sandbox

However, this may involve the interactions with multiple Site objects, which hasn't been fully considered yet. Plus, I'm not sure whether the latter handling is necessary.

Flow.Board.EnumTopicsAsync failure internal_api_error_Flow\Exception\NoParserException

MW API endpoint: https://en.wikipedia.beta.wmflabs.org/w/api.php
- MW version: 1.35.0-alpha (b02b270)
- Flow version: 1.1.1 (c4ab042)

Stack trace

WikiClientLibrary.MediaWikiRemoteException: internal_api_error_Flow\Exception\NoParserException: [XZ9XTawQBGoAAB-47awAAAAS] Exception caught: Request to parsoid for "html" to "wikitext" conversion of content connected to title "Topic:V68sl78cnj95fsls" failed: (curl error: 28) Timeout was reached
Flow\Exception\NoParserException at /srv/mediawiki/php-master/extensions/Flow/includes/Conversion/Utils.php(174)
#0 /srv/mediawiki/php-master/extensions/Flow/includes/Conversion/Utils.php(66): Flow\Conversion\Utils::parsoid(string, string, string, Title)
#1 /srv/mediawiki/php-master/extensions/Flow/includes/Model/AbstractRevision.php(430): Flow\Conversion\Utils::convert(string, string, string, Title)
#2 /srv/mediawiki/php-master/extensions/Flow/includes/Templating.php(160): Flow\Model\AbstractRevision->getContent(string)
#3 /srv/mediawiki/php-master/extensions/Flow/includes/Formatter/RevisionFormatter.php(289): Flow\Templating->getContent(Flow\Model\PostRevision, string)
#4 /srv/mediawiki/php-master/extensions/Flow/includes/Formatter/TopicListFormatter.php(71): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, DerivativeContext)
#5 /srv/mediawiki/php-master/extensions/Flow/includes/Formatter/TopicListFormatter.php(48): Flow\Formatter\TopicListFormatter->buildResult(array, array, DerivativeContext)
#6 /srv/mediawiki/php-master/extensions/Flow/includes/Block/TopicListBlock.php(298): Flow\Formatter\TopicListFormatter->formatApi(Flow\Model\Workflow, array, array, Flow\Data\Pager\PagerPage, DerivativeContext)
#7 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlowBaseGet.php(33): Flow\Block\TopicListBlock->renderApi(array)
#8 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlow.php(105): Flow\Api\ApiFlowBaseGet->execute()
#9 /srv/mediawiki/php-master/includes/api/ApiMain.php(1588): Flow\Api\ApiFlow->execute()
#10 /srv/mediawiki/php-master/includes/api/ApiMain.php(537): ApiMain->executeAction()
#11 /srv/mediawiki/php-master/includes/api/ApiMain.php(508): ApiMain->executeActionWithErrorHandling()
#12 /srv/mediawiki/php-master/api.php(83): ApiMain->execute()
#13 /srv/mediawiki/w/api.php(3): require(string)
#14 {main}
--- End of MediaWiki remote stack trace ---
   at WikiClientLibrary.Client.MediaWikiJsonResponseParser.OnApiError(String errorCode, String errorMessage, JToken errorNode, JToken responseNode, WikiResponseParsingContext context) in $\\WikiClientLibrary\Client\MediaWikiJsonResponseParser.cs:line 183
   at WikiClientLibrary.Client.MediaWikiJsonResponseParser.ParseResponseAsync(HttpResponseMessage response, WikiResponseParsingContext context) in $\\WikiClientLibrary\Client\MediaWikiJsonResponseParser.cs:line 84
   at WikiClientLibrary.Client.WikiResponseMessageParser`1.WikiClientLibrary.Client.IWikiResponseMessageParser<T>.ParseResponseAsync(HttpResponseMessage response, WikiResponseParsingContext context) in $\\WikiClientLibrary\Client\WikiResponseMessageParser.cs:line 64
   at WikiClientLibrary.Client.WikiClient.SendAsync[T](String endPointUrl, WikiRequestMessage message, IWikiResponseMessageParser`1 responseParser, CancellationToken cancellationToken) in $\\WikiClientLibrary\Client\WikiClient.cs:line 265
   at WikiClientLibrary.Client.WikiClient.SendAsync[T](String endPointUrl, WikiRequestMessage message, IWikiResponseMessageParser`1 responseParser, CancellationToken cancellationToken) in $\\WikiClientLibrary\Client\WikiClient.cs:line 281
   at WikiClientLibrary.Client.WikiClient.InvokeAsync[T](String endPointUrl, WikiRequestMessage message, IWikiResponseMessageParser`1 responseParser, CancellationToken cancellationToken) in $\\WikiClientLibrary\Client\WikiClient.cs:line 155
   at WikiClientLibrary.Sites.WikiSite.InvokeMediaWikiApiAsync[T](WikiRequestMessage message, IWikiResponseMessageParser`1 responseParser, Boolean suppressAccountAssertion, CancellationToken cancellationToken) in $\\WikiClientLibrary\Sites\WikiSite.cs:line 386
   at WikiClientLibrary.Flow.Board.<>c__DisplayClass20_0.<<EnumTopicsAsync>b__0>d.MoveNext() in $\\WikiClientLibrary.Flow\Board.cs:line 127
--- End of stack trace from previous location where exception was thrown ---
   at AsyncEnumerableExtensions.TaskAsyncEnumerable`1.Enumerator.PropagateGeneratorException()
   at AsyncEnumerableExtensions.TaskAsyncEnumerable`1.Enumerator.MoveNextAsync()
   at System.Linq.AsyncEnumerablePartition`1.ToArrayAsync(CancellationToken cancellationToken) in d:\a\1\s\Ix.NET\Source\System.Linq.Async\System\Linq\AsyncEnumerablePartition.cs:line 300
   at System.Linq.AsyncEnumerablePartition`1.ToArrayAsync(CancellationToken cancellationToken) in d:\a\1\s\Ix.NET\Source\System.Linq.Async\System\Linq\AsyncEnumerablePartition.cs:line 321
   at WikiClientLibrary.Tests.UnitTestProject1.Tests.FlowTests.BoardTest() in $\\UnitTestProject1\Tests\FlowTests.cs:line 44
--- End of stack trace from previous location where exception was thrown ---

Flow (aka Structured Discussions)

I'm trying to add Flow API support to the library, like what has been done in PyWikiBot
.

https://www.mediawiki.org/wiki/Extension:Flow

support ?action=raw

Hi, I'm not sure if I'm missing something or not, but I couldn't find a way to return the raw content of a page.
The only thing I thought was remotely relevant is AbuseFilter but I couldn't understand how to use it...
Here's a link to an example:
https://en.wikipedia.org/wiki/Main_Page?action=raw

Remove WikiSite.CreateAsync to make it inheritance-friendly

I'm going to remove WikiSite.CreateAsync, and replace it with a kind of two step initialization, like the IAsyncInitialization in Stephen's Async OOP 2: Constructors.

var site = new WikiSite(client);
await site.Initialization;

so that I can derive new types easily from WikiSite class.

Background

Most of the functionality (except for LoginAsync and LogoutAsync, actually) in WikiSite depend on the site information as well as user information, which need to be fetched asynchronously beforehand.

When I first built the class, I thought of making the states in this class valid, and what I needed is an asynchronous constructor. This is impossible. (Though there are now some discussions at dotnet/csharplang#419 on async ctors) I don't want to write something like

public WikiSite(....)
{
    // Initializations ....
    Task.WhenAll(site.RefreshSiteInfoAsync(), site.RefreshAccountInfoAsync()).Wait();
}

So it came naturally to me that perhaps I can delegate the construction process into a separate asynchronous factory method, and that's what we have now.

As I have noticed, and been bothered constantly, is that using factory methods to publish a type makes it difficult (or at least, troublesome) to write derived classes for it. But at that early time, there is no need to derive from WikiSite class, and I even thought about making it sealed. As I have mentioned in #18:

there is only one WikiSite class in the whole library, providing the basic funtionality: construct a WikiSite instancce.

If you have your own WikiSite-derived class (which is not so often, I suppose), you may write your own static factory method in another class.

However, with the advent of Wikia API implementation, I introduced WikiaSite. To keep the new class neat, I made it independent from any WikiSite instance. However, the day finally come when I need to access to MediaWiki API token to perform Wikia-specific edit operations. I need a TokensManager instance, which only exists in WikiSite.

It made me to re-think about inheriting from WikiSite class. I could maintain an instance of WikiSite class as a property of WikiaSite, it would look ugly. WikiaSite will rely on WikiSite's IWikiClient, TokensManager, and perhaps other instance we don't know for now. The coupling is strong enough for inheritance.

On the other hand, Wikia sites are only a kind of specialized MediaWiki sites, so it will be natural, from client's perspective, to use WikiaSite as a subclass of WikiSite. As Pzixel has observed in the issue in csharplang

But if we are going to make a factory method, we just can't inherit this object, or we should copy-paste CreateAsync method or share it in some other way between descendants. The best we can do is use composition instead of inheritance. But sometimes it breaks is rule - if we are saying A is B then A should inherit B, but call its CreateAsync method.

Wikibase API

See mw:Wikibase/API.

This is a brand-new set of APIs to implement. May work in slow progress. (See #5 for a rough idea of such slowness 🌚)

I will work it on another branch now and then, when I do not feel like working on master.

Extract list from generator

Currently, all the lists are encapsulated as generators. For certain cases, this is unnecessary or even undesirable (e.g. recent activities, users).

I will try to extract a WikiList class from the current WikiGenerator, and in the future, WikiGenerator would less often needs to be derived from.

Connecting to API is failing with a time-out

Firstly, thank you for writing this library.

Today my Discord bot started failing to link [[wiki links]] with their respective articles. In the code I use this library (v. 0.5.1). The cause of this, as I understand after running testing versions, is because of a time-out from Site.CreateAsync.

I’ve tried to update the library to last pre-release version internally, but this error is still appearing and I don’t understand the cause. The error wasn’t present before today and Wikipedia API (specifically ru.wikipedia.org) seems to work fine. The stack trace for both versions is nearly the same, but I’ll publish it here for the convenience.

Sorry in advance if I am wasting your time and the error is on my side and not in the library.

0.5.1:

System.AggregateException was unhandled
  HResult=-2146233088
  Message=One or more errors occurred.
  Source=mscorlib
  StackTrace:
       at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
       at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at System.Threading.Tasks.Task`1.get_Result()
       at DiscordWikiBot.Linking.Init() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Linking.cs:line 37
       at DiscordWikiBot.Program.<Run>d__8.MoveNext() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Program.cs:line 71
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
       at DiscordWikiBot.Program.Main(String[] args) in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Program.cs:line 27
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 
       HResult=-2146233083
       Message=The operation has timed out.
       Source=WikiClientLibrary
       StackTrace:
            at WikiClientLibrary.Client.WikiClient.<SendAsync>d__0.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Client.WikiClient.<GetJsonAsync>d__19.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Site.<PostValuesAsync>d__44.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Site.<RefreshSiteInfoAsync>d__16.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Site.<CreateAsync>d__13.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
            at DiscordWikiBot.Linking.<FetchSiteInfo>d__8.MoveNext() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Linking.cs:line 215
       InnerException:

0.6-int6:

System.AggregateException was unhandled
  HResult=-2146233088
  Message=One or more errors occurred.
  StackTrace:
       at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
       at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
       at System.Threading.Tasks.Task`1.get_Result()
       at DiscordWikiBot.Linking.Init() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Linking.cs:line 38
       at DiscordWikiBot.Program.<Run>d__8.MoveNext() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Program.cs:line 71
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
       at DiscordWikiBot.Program.Main(String[] args) in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Program.cs:line 27
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: 
       HResult=-2146233083
       Message=The operation has timed out.
       StackTrace:
            at WikiClientLibrary.Client.WikiClient.<SendAsync>d__38`1.MoveNext()
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Client.WikiClient.<InvokeAsync>d__35`1.MoveNext()
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
            at WikiClientLibrary.Sites.WikiSite.<InvokeMediaWikiApiAsync>d__47`1.MoveNext()
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Sites.WikiSite.<RefreshSiteInfoAsync>d__21.MoveNext()
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at WikiClientLibrary.Sites.WikiSite.<>c__DisplayClass20_0.<<-ctor>g__InitializeAsync|1>d.MoveNext()
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
            at DiscordWikiBot.Linking.<FetchSiteInfo>d__8.MoveNext() in C:\Users\####\documents\visual studio 2015\Projects\DiscordWikiBot\DiscordWikiBot\Linking.cs:line 217
       InnerException:

Coordinate query fails on some thread cultures

Since the GeoSearchGenerator uses standard ToString for coordinate latitudes and longitudes, the api returns invalid coordinates, if the current thread culture is set to a culture, where decimal places are separated using a comma "," instead of a point ".".
I suggest using the ToString overload with a CultureIdentifier and using InvariantCulture in all relevant places.

RecentChangesGenerator#EnumRecentChanges and RecentChangesGenerator#EnumRecentChangesAsync don't finish

My code freezes at this location in my code. I have no idea why and i have no idea where to post this.
The api endpoint: "http://gamemode4.wikia.com/api.php"

The first breakpoint gets triggered. The second one wont

Occasional "badtoken:Invalid CSRF token" error

IsraelHikingMap/Site#481 is about occasional failure to upload pictures to Wikimedia using WikiClientLibrary.

Many uploads succeeded, but from time to time, the there is a "badtoken:Invalid CSRF token" exception. For example:

2017-09-22 21:33:37.7688 | 13|ERROR| soft.AspNetCore.Server.Kestrel | Connection id "0HL81JJA3C9J5": An unhandled exception was thrown by the application. WikiClientLibrary.OperationFailedException: badtoken:Invalid CSRF token.
   at WikiClientLibrary.Client.WikiClient.CheckErrors(JToken jresponse)
   at WikiClientLibrary.Client.WikiClient.<SendAsync>d__0.MoveNext()

What is this error?
How can it be avoided?

Get page by id - simplified

I was browsing thorough the code but could not find a way to get a page by its page Id.
This can be probably be achieved by query generator but it seems a bit too complicated for what I believe should be part of the WikiPage constructor/factory.
Am I missing something?

Getting a strange error while initializing wikisite

Unexpected character encountered while parsing value: <. Path '', line 0, position 0.

Running the following code:

var wikiClient = new WikiClient
            {
                ClientUserAgent = "IsraelHikingMapSite/5.x",
                Timeout = TimeSpan.FromMinutes(1)
            };
            _wikiSite = new WikiSite(wikiClient, new SiteOptions("http://inature.info/w/api.php"));
            await _wikiSite.Initialization;

Not sure what I should do.
Probably an issue with the site I'm using, but still...

Migrate unit tests to xUnit

… which enables some real asynchronous test methods.

Update documentation

Hi,

Can you please update documentation?
The following is incorrect:
var site = await WikiSite.CreateAsync(wikiClient, "https://test2.wikipedia.org/w/api.php");
Should be (I hope)
var site = await Site.CreateAsync(wikiClient, "https://test2.wikipedia.org/w/api.php");

Also would be helpful to provide more example of API usage, for example, I want to use this library to upload images, but I'm not sure how to do it from the documentation.

‘infinite’ is an invalid DateTime

Hey, first of all, great library :-)

Some wikis (most of them?) use infinite in block expiration options, among them are Dutch Wikipedia and Wikitech. Fetching SiteInfo with this library with an active block from those wikis (for me, it is a block for hosting range) returns a FormatException in my bot.

I suspect that there can also be indefinite value triggering the same error, but I don’t have administrative tools anywhere to test the theory.

Fix should be pretty simple: adding infinite as an alias to infinity where applicable. This will close stjohann/DiscordWikiBot#3 for me.

Shortened trace log:

[2019-07-15 22:16:19] [DiscordWikiBot] [Error] Exception occurred: System.AggregateException: Exceptions occured within one or more event handlers. Check InnerExceptions for details. (One or more errors occurred. (The string 'infinite' was not recognized as a valid DateTime. There is an unknown word starting at index '0'.)) ---> System.AggregateException: One or more errors occurred. (The string 'infinite' was not recognized as a valid DateTime. There is an unknown word starting at index '0'.) ---> System.FormatException: The string 'infinite' was not recognized as a valid DateTime. There is an unknown word starting at index '0'.
   at System.DateTimeParse.Parse(ReadOnlySpan`1 s, DateTimeFormatInfo dtfi, DateTimeStyles styles)
   at System.DateTime.Parse(String s, IFormatProvider provider, DateTimeStyles styles)
   at WikiClientLibrary.Infrastructures.WikiDateTimeJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
...

Need documentation to query an entity

I am looking for a way to query an entity using the name and finding the QID for it. Can someone point me to an example? Thanks!

Here is what I have so far. I am able to fetch an entity using QID, but I am looking to fetch an entity by it's name.

// A WikiClient has its own CookieContainer.
var client = new WikiClient
{
ClientUserAgent = "WCLQuickStart/1.0 (your user name or contact information here)"
};

var myWikiDataSite = new WikiSite(client, "https://www.wikidata.org/w/api.php");
try
{
await myWikiDataSite.LoginAsync(Username, Password);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}

try
{
await myWikiDataSite.Initialization;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}

var entity = new Entity(myWikiDataSite, "Q60204555");
await entity.RefreshAsync( EntityQueryOptions.FetchAllProperties);

await myWikiDataSite.LogoutAsync();
client.Dispose();
return;

MediaWikiHelper.MakeAbsoluteUrl protocol not so intuitive

The following call raises an exception:
I wanted to get a full URL of a page (I'm sure there is a simpler way but I couldn't find it...)

var allpagesGenerator = new AllPagesGenerator(_wikiSite)
{
    PaginationSize = 500
};
var results = await allpagesGenerator.EnumItemsAsync().ToList();
var page = new WikiPage(_wikiSite, results.First().Title);
var pageRelativeUrl = _wikiSite.SiteInfo.ArticlePath.Replace("$1", page.Title);
MediaWikiHelper.MakeAbsoluteUrl(_wikiSite.SiteInfo.ServerUrl + "/wiki", pageRelativeUrl, "http");

Apparently I need to send "http:" as protocol string, which I find not intuitive.

GeoSearchGenerator does not support query continuation

Currently, GeoSearchGenerator does not support query continuation, because MW API action=query&list=geosearch does not support so. This is tracked by phab:T95241 and phab:T78703.

Example response of https://en.wikipedia.org/w/api.php?action=query&maxlag=5&list=geosearch&gsradius=10&gsprimary=primary&gslimit=2&gsbbox=32.15%7C34.75%7C32%7C34.9

{
    "batchcomplete": "",
    "query": {
        "geosearch": [
            {
                "pageid": 18328987,
                "ns": 0,
                "title": "Beit Zvi",
                "lat": 32.078408333333336,
                "lon": 34.821713888888894,
                "dist": 489.4,
                "primary": ""
            },
            {
                "pageid": 46324352,
                "ns": 0,
                "title": "HaAliya HaShniya Garden",
                "lat": 32.0697,
                "lon": 34.8148,
                "dist": 1127.4,
                "primary": ""
            }
        ]
    }
}

Better upload API

When using the wikimedia common upload wizard the following fields are editable:

Tags
Geo Location
Description
Date
Would be nice if the client FilePage.Upload would support those as well.
UploadResults object has a fileKey which is good in order to get the file etc, but a full address to the file would be helpful too, as I guess in most cases people would use the link to present the image.

"Note that the pages are sorted in alphabetical order, so the second item might be something like Q10 rather than Q10"

Shouldn't it be Q2?

MakeArticleUrl ignores defaultProtocol?

I'm using wikiSite.SiteInfo.MakeArticleUrl("some title", "http:") but I'm getting back an address with https, am I missing something?
wikisite init: wikiSite = new WikiSite(wikiClient, new SiteOptions("http://inature.info/w/api.php"))

Throttler does not prevent additional threads from performing parallel requests

While debugging I noticed that the built in throttler is simply a delay before any 'edit' tier modification is performed, meaning that numerous threads could end up submitting changes to a MediaWiki API at the same time.

Additionally, FilePage.UploadAsync does not utilise throttling at all.

It should be possible to implement an inter-thread throttler without introducing any breaking changes.

Private MediaWiki not accessable

I have a private MediaWiki that is not possible to access with WikiClientLibrary

The MediaWiki have following settings:

$wgGroupPermissions['*']['read'] = false;
$wgGroupPermissions['*']['edit'] = false;
$wgGroupPermissions['*']['createaccount'] = false;

Dealing with invalid dates

On certain wiki pages, such as "Elon Musk", the following error occurs when executing
await page.RefreshAsync(PageQueryOptions.FetchContent | PageQueryOptions.ResolveRedirects);
On a page (Note the code i am using works on all pages except from a few, such as the above mentioned). Error:
Newtonsoft.Json.JsonSerializationException: Error setting value to 'ExpiryProxy' on 'WikiClientLibrary.Pages.ProtectionInfo'. ---> System.FormatException: String was not recognized as a valid DateTime.

Login with clientlogin

This library aims to provide functionality for both bots and users, so it's important, especially in the future, to implement clientlogin action.

See https://www.mediawiki.org/wiki/API:Login#The_clientlogin_action .

"Value was either too large or too small for an Int32."

I'm getting a Int32OverflowExeption when accessing Wikia through your client.
Here's my code:

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Discord;
using Discord.WebSocket;
using System.Linq;
using WikiClientLibrary.Client;
using WikiClientLibrary.Sites;
using WikiClientLibrary.Pages;
using WikiClientLibrary.Wikia;
using Discord.Commands;
using Microsoft.Extensions.DependencyInjection;
using System.Reflection;
using WikiClientLibrary.Wikia.Sites;

namespace EC_Discord_bot_CS
{
    class Program
    {
        public static WikiaSite site;

        private CommandService commands;

        private IServiceProvider services;

        private DiscordSocketClient discordClient;

        static void Main(string[] args)
        {
            new Program().MainAsync().Wait();
        }

        public async Task MainAsync()
        {
            #region wikia client
            var wikiaClient = new WikiClient
            {
                ClientUserAgent = "Edge-bot"
            };

            try
            {
                WikiaSiteOptions options = new WikiaSiteOptions()
                {
                    ApiEndpoint = "http://theedgechronicles.wikia.com/api.php",
                    WikiaApiRootUrl = "http://theedgechronicles.wikia.com/api/v1",
                };
                site = new WikiaSite(wikiaClient, options, "username", "password"); //yes, I used my real username and password
                await site.Initialization; //Exception here
            }
            catch (Exception EX)
            {
                Console.WriteLine(EX.Message);
            }
            #endregion
     }
}

Here's the Exception message:

{System.OverflowException: Value was either too large or too small for an Int32.
   at System.Convert.ThrowInt32OverflowException()
   at System.Convert.ToInt32(Int64 value)
   at Newtonsoft.Json.JsonReader.ReadAsInt32()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JToken.ToObject(Type objectType)
   at Newtonsoft.Json.Linq.JToken.ToObject[T]()
   at WikiClientLibrary.Wikia.Sites.WikiaSite.<RefreshSiteInfoAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at WikiClientLibrary.Sites.WikiSite.<>c__DisplayClass18_0.<<-ctor>g__InitializeAsync|1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at EC_Discord_bot_CS.Program.<MainAsync>d__5.MoveNext() in C:\Users\ADMIN\Documents\Visual Studio 2017\Projects\EC Discord bot CS\EC Discord bot CS\Program.cs:line 53}

I'm stumped as to how to fix this, as the Exception seems to coming from a value that Newtonsoft tried to parse to an Int32 during the Initialization call.

API:Search seems lacking for support

I've implemented SearchGenerator per API:Search. However, I got search-title-disabled error when srwhat is title on Wikipedia, and got search-***-disabled for whatever srwhat on Wikia.

Even though the search works on Wikipedia, I still cannot get the same order as the search result shown on Special:Search page, with the same keyword.

UriFormatException when writing a large page

The exception was raised when I was writing a page with the content of 77KB.

Unhandled Exception: System.AggregateException: One or more errors occurred. (Invalid URI: The Uri string is too long.) ---> System.UriFormatException: Invalid URI: The Uri string is too long.
   at System.UriHelper.EscapeString(String input, Int32 start, Int32 end, Char[] dest, Int32& destPos, Boolean isUriString, Char force1, Char force2, Char rsvd)
   at System.Uri.EscapeDataString(String stringToEscape)
   at System.Net.Http.FormUrlEncodedContent.Encode(String data)
   at System.Net.Http.FormUrlEncodedContent.GetContentByteArray(IEnumerable`1 nameValueCollection)
   at System.Net.Http.FormUrlEncodedContent..ctor(IEnumerable`1 nameValueCollection)
   at WikiClientLibrary.Client.WikiClient.<>c__DisplayClass40_0.<GetJsonAsync>b__0()
   at WikiClientLibrary.Client.WikiClient.<SendAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at WikiClientLibrary.Client.WikiClient.<GetJsonAsync>d__40.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at WikiClientLibrary.Page.<UpdateContentAsync>d__100.MoveNext()

It seems to be the limitation of System.Uri.EscapeDataString. Perhaps I need to re-implement it.

Contest

In the wikia dev discord they announced a contest for MW libraries.

Rules: https://gist.github.com/KockaAdmiralac/5465c7c49f381291cd740c261ea0d89c

I asked if existing projects could compete too.
They can as long as they arent more then 2 years old.

Prize:

In the form of
steam credit
Vpn credit
Or discord nitro

FilePage.UploadAsync does not have retry logic

While performing some automatic page creation, I noticed that there were clusters of file upload failures. These failure clusters aligned with short periods of poor WiFi connectivity, and investigating the library code revealed that there was no retry logic, meaning that any dip in connectivity can result in several uploads failing.

DeleteAsync and MoveAsync fail if Site.Logger is null, AND ModificationThrottler will hang for all further requests

These async methods fail if Site.Logger is null
Site.Logger.Info(this, $"Page {title} has been deleted."); // assumes that Site.Logger is not null
Site.Logger?.Info(this, $"Page {title} has been deleted."); // should be implemented as shown to the left (2 locations)
If a task fails, the task is not marked as completed.
The ModificationThrottler for the next task will wait for the previous task to complete.
As the failed task will never complete, ModificationThrottler will be in an endless hang.

Strange error message

I'm getting the following error message:

> (Inner Exception #4) System.AggregateException: One or more errors occurred. (assertuserfailed:Assertion that the user is logged in failed.) ---> WikiClientLibrary.AccountAssertionFailureException: assertuserfailed:Assertion that the user is logged in failed.
   at WikiClientLibrary.Client.MediaWikiJsonResponseParser.OnApiError(String errorCode, String errorMessage, JToken errorNode, JToken responseNode, WikiResponseParsingContext context)
   at WikiClientLibrary.Client.MediaWikiJsonResponseParser.<ParseResponseAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---

I'm not sure why I got this error message but restarting the process solved it.
Is there an expiration to user credentials? Should I be expecting this exception and heal from it somehow?
Any help would be appreciated! :-)

Add support for bounding box in GeoSearchGenerator

The Wikimedia geosearch supports the use of a bounding box as an alternative to the coordinates+radius as a Geograpic selector:

gsbbox: Bounding box to search in: pipe (|) separated coordinates of top left and bottom right corners.

and provides an example:

api.php?action=query&list=geosearch&gsbbox=37.8|-122.3|37.7|-122.4

Since the coordinates+radius approach is limited to a 10000 meter radius, combining multiple requests in order to cover a larger area is a challenge. On the other hand, the use of a bounding box for searching Wikimedia is easier to aggregate and to integrate with other Geographic systems

Please consider adding support for search based on a bounding box.

Not a bug, just a question

Hi,

No issues that I've encountered I just had a few questions and couldn't find a method to contact you via email.

In short, if you could reach out to me I'd sincerely appreciate it. @CXuesong email is [email protected]

Introduce SourceLink support

c.f. https://github.com/dotnet/sourcelink
This enables you, as library users, to debug into the source code of WikiClientLibrary in VS 2017+. Source code should be downloaded on demand from GitHub when debugging.

I chose to include debugging information in the NuGet packages since v0.6.0 (or b9b3a43 to be more exact). Hope this facilitates the your debugging procedure.

If you come across any problems when debugging into WCL library, please report your problems in this issue.

Use an existing logging framework for the activity logs

As mentioned in #13, the current WikiClientLibrary.ILogger is just error-prone for the library-side code, and the class name is very likely to conflict with other logging libraries.
May consider Microsoft.Extensions.Logging.Abstractions, though Serilog might also be a good choice.

A good point might be that if you pass a ILoggerFactory into WikiClient, this factory can be used automatically ~~(or not, configured by user)~~[1] in other related classes, e.g., WikiSites and WikiClients.

[1] The user can manually remove the logger after the related class instance is created.

Namespace, name and other API adjustments in v0.6

Maybe it's time to reorganize the classes now. More detail follows.

Generalize PageQueryOptions

For now, client code can use this enum to specify whether to fetch for content of the page, and/or to resolve the redirects when querying for the page. But this degree of customization is limited. If the client wants to fetch for the extract for a page, they might want some further customization on how many sentences per page they want the server to return; if the client wants to fetch for the page image for a page, they might want to specify the dimension of the thumbnails. To enable fetching extracts, page images, or more extendable properties from MW server with some more allowance for customization, we might need to replace PageQueryOptions enum with a class that contains enough information to compose parameters for action=query requests.

By generalizing PageQueryOptions, we can also replace the current MediaWikiHelper.GetPageQueryParameters static method with some more publicized & configurable parameters generators.

The final goal is to make an extendable interface that, when implemented, allows client code to request for any page property xxx as in action=query&prop=xxx. E.g. xxx can be revisions, extracts, etc.

A strange behavior when Enumerable.Select is chained with AsyncEnumerable.Take

Test case

public class Playground : UnitTestsBase
{

    /// <inheritdoc />
    public Playground(ITestOutputHelper output) : base(output)
    {
    }

    [Fact]
    public async Task Test1()
    {
        var invoked = false;
        var ienu = new DelegateAsyncEnumerable<IEnumerable<string>>(async ct =>
        {
            if (invoked) return null;
            await Task.Yield();
            IEnumerable<string> x = new[] {"abc", "def", "ghi", "jkl"};
            invoked = true;
            return Tuple.Create(x.Select(a => a + "a"), true);
        });
        var y = ienu.SelectMany(t => t.ToAsyncEnumerable());
        var z = await y.Take(4).ToArray();
        ShallowTrace(z);
        Assert.DoesNotContain(null, z);
    }

}

The assertion fails. If I either change Take(4) to Take(3), change Select invocation to x.Select((a, i) => a + "a"), or remove await Task.Yield(); line, the test passes.

It seems that there is a strange interaction between the underlying SelectArrayIterator<TSource, TResult> and thread-context switches. I got a rough idea when looking at Iterator<TSource>.GetEnumerator, but would take another time to dig it deeper.

Related code in library:

WikiClientLibrary/WikiClientLibrary/Flow/Board.cs

Lines 108 to 111 in 435681d

  // ISSUE directly return SelectArrayIterator<TSource, TResult> via Enumerable.Select 

  // can cause strange behavior when chained with other async LINQ methods. See 

  // https://github.com/CXuesong/WikiClientLibrary/issues/27 

  var topics = Topic.FromJsonTopicList(Site, jtopiclist).ToList();

action=logout and Wikia

As indicated by a recent regression testing, action=logout has been removed from current Wikia release of MediaWiki API. This will cause the following exception when calling WikiSite.LogoutAsync for all Wikia sites on FANDOM.

WikiClientLibrary.InvalidActionException : unknown_action:Unrecognized value for parameter 'action': logout

From the information gathered, there is no pure MW API approach to just logout a user, and some workaround is inevitable.

c.f. Server-side logout procedure: https://github.com/Wikia/app/blob/4988ecbd6938a90f6f66fbc33cf31d63e720adcc/includes/User.php#L3155 (Thanks to fngplg)

Reduce GetTokensAsync to GetTokenAsync

When I first wrote Site.GetTokensAsync, I had assumed that a user might want to retrieve different tokens in one MW API request. But this assumption led to too much complexity. On the other hand, its overload in singular form GetTokenAsync is used much more often compared with GetTokensAsync.

To reduce the complexity and make it easier to maintain, I'm going to remove GetTokensAsync. If, in the rare case, I assume, you want to retrieve multiple tokens, you will need to call GetTokenAsync multiple times.

Prefer interfaces instead of static factories

The following is impossible to fake in tests:
Site.CreateAsync(...)
which means that if I want to using this client in my code and be able to test my code without doing calls to wiki I'll need to wrap it with my code, which, IMHO, kinds of misses the point a bit.
A better API would provide factories for all the object so that I'll be able to mock the factory in my tests.
For the example above the following would do wonders when trying to test code that uses this library:

public class SiteFactory : ISiteFactory {
    public Task<Site> CreateAsync(...) { ... }
}

I don't mind helping out with the relevant code changes using pull requests, but I need an initial approval that these changes are acceptable.

Include/add headers to WikiSite request

I'm trying to access a wiki that uses SAML authentication. Is there a method to include/add headers to a the WikiSite request?

Need a MediaWiki site to run dirty tests

I've written test cases for page moving in PageTestsDirty.cs. In future I'll write test cases for deleting and other operations.

The problem is, I'm only a user on http://test2.wikipedia.org/ , so I need to make a request for bot on the site, or find another site before I can run the tests.

I just thought it may take up too much time if I set up a site on my own.

Chunked uploading

See mw:API:Upload#Chunked uploading.

This can be helpful especially for uploading big files without triggering time-outs.

However, a dedicated "uploader" non-static class might be needed, considering now FilePage.UploadAsync has a lot of parameters and in the future the uploading process would be stateful.

	// ISSUE directly return SelectArrayIterator<TSource, TResult> via Enumerable.Select
	// can cause strange behavior when chained with other async LINQ methods. See
	// https://github.com/CXuesong/WikiClientLibrary/issues/27
	var topics = Topic.FromJsonTopicList(Site, jtopiclist).ToList();