Hey @macmade, I just discovered this project and was really delighted by how easy it was to start using to parse Objective-C files. I understand that you haven't used it in a while, and have moved onto other projects, but I was curious if you had any idea how much work it would be to support reading swift files and getting back an array of tokens?

Currently, the call to -tokens returns nil, so it would seem that the call to clang_tokenize yields zero tokens.

When makefile run into line
'@$(CD) $(DIR_BUILD) && ../$(DIR_LLVM)configure --enable-optimized'
it will output:

'The LLVM project no longer supports building with configure & make.
Please migrate to the CMake-based build system.
For more information see: http://llvm.org/docs/CMake.html'.

I guess maybe the version of 'ClangKit/LLVM/Source/llvm' is not latest?

Your installation guide instructs to build the LLVM sources using the makefile.
After running the makefile, what must be done?

A use case I have for ClangKit is, given a file, get the translation unit for it and then parse the stream of tokens into a tree-like structure. This requires getting a valid cursor for the tokens, so I can identify class and method declarations. Often, I just have a .m or .mm file, and calling [CKTranslationUnit translationUnitWithPath:somePath] does the right thing.

However, if I call [CKTranslationUnit translationUnitWithPath:@"/my/special/file.h"] then the actual tokens returned say the cursor is "NoDeclFound" for the entire file. You can see this below:

(lldb) po translationUnit.tokens
<__NSArrayI 0x6000001c0000>(
<CKToken: 0x600000085b40>: Punctuation[1:1] # (inclusion directive),
<CKToken: 0x600000085b90>: Identifier[1:2] import (inclusion directive),
<CKToken: 0x600000085be0>: Punctuation[1:9] < (inclusion directive),
<CKToken: 0x600000085c30>: Identifier[1:10] Foundation (NoDeclFound),
<CKToken: 0x600000085c80>: Punctuation[1:20] / (NoDeclFound),
<CKToken: 0x600000085cd0>: Identifier[1:21] Foundation (NoDeclFound),
<CKToken: 0x600000085d20>: Punctuation[1:31] . (NoDeclFound),
<CKToken: 0x600000085d70>: Identifier[1:32] h (NoDeclFound),
<CKToken: 0x600000085dc0>: Punctuation[1:33] > (NoDeclFound),
<CKToken: 0x600000085e10>: Punctuation[2:1] @ (NoDeclFound),
<CKToken: 0x600000085e60>: Identifier[2:2] interface (NoDeclFound),
<CKToken: 0x600000085eb0>: Identifier[2:12] XMASAlert (NoDeclFound),
<CKToken: 0x600000085f00>: Punctuation[2:22] : (NoDeclFound),
<CKToken: 0x600000085f50>: Identifier[2:24] NSObject (NoDeclFound),
<CKToken: 0x600000085fa0>: Punctuation[3:1] - (NoDeclFound),
<CKToken: 0x600000085ff0>: Punctuation[3:3] ( (NoDeclFound),
<CKToken: 0x600000086040>: Keyword[3:4] void (NoDeclFound),
<CKToken: 0x600000086090>: Punctuation[3:8] ) (NoDeclFound),
<CKToken: 0x6000000860e0>: Identifier[3:9] flashMessage (NoDeclFound),
<CKToken: 0x600000086130>: Punctuation[3:21] : (NoDeclFound),
<CKToken: 0x600000086180>: Punctuation[3:22] ( (NoDeclFound),
<CKToken: 0x6000000861d0>: Identifier[3:23] NSString (NoDeclFound),
<CKToken: 0x600000086220>: Punctuation[3:32] * (NoDeclFound),
<CKToken: 0x600000086270>: Punctuation[3:33] ) (NoDeclFound),
<CKToken: 0x6000000862c0>: Identifier[3:34] message (NoDeclFound),
<CKToken: 0x600000086310>: Punctuation[3:41] ; (NoDeclFound),
<CKToken: 0x600000086360>: Punctuation[4:1] @ (NoDeclFound),
<CKToken: 0x6000000863b0>: Identifier[4:2] end (NoDeclFound)
)

I have found a workaround of calling CKTranslationUnit translationUnitWithText: language:, which actually does provide the correct cursor for the header. This looks a bit like this:

(lldb) po translationUnit.tokens
<__NSArrayI 0x6000001c12c0>(
<CKToken: 0x600000085910>: Punctuation[1:1] # (inclusion directive),
<CKToken: 0x6000000859b0>: Identifier[1:2] import (inclusion directive),
<CKToken: 0x600000085730>: Punctuation[1:9] < (inclusion directive),
<CKToken: 0x600000085690>: Identifier[1:10] Foundation (NoDeclFound),
<CKToken: 0x6000000856e0>: Punctuation[1:20] / (NoDeclFound),
<CKToken: 0x6000000855f0>: Identifier[1:21] Foundation (NoDeclFound),
<CKToken: 0x600000085640>: Punctuation[1:31] . (NoDeclFound),
<CKToken: 0x600000085550>: Identifier[1:32] h (NoDeclFound),
<CKToken: 0x600000085500>: Punctuation[1:33] > (NoDeclFound),
<CKToken: 0x6000000855a0>: Punctuation[2:1] @ (ObjCInterfaceDecl),
<CKToken: 0x6000000854b0>: Keyword[2:2] interface (ObjCInterfaceDecl),
<CKToken: 0x600000085460>: Identifier[2:12] XMASAlert (ObjCInterfaceDecl),
<CKToken: 0x600000085410>: Punctuation[2:22] : (ObjCInterfaceDecl),
<CKToken: 0x6000000853c0>: Identifier[2:24] NSObject (ObjCSuperClassRef),
<CKToken: 0x600000085320>: Punctuation[3:1] - (ObjCInstanceMethodDecl),
<CKToken: 0x600000085370>: Punctuation[3:3] ( (ObjCInstanceMethodDecl),
<CKToken: 0x600000085280>: Keyword[3:4] void (ObjCInstanceMethodDecl),
<CKToken: 0x600000085230>: Punctuation[3:8] ) (ObjCInstanceMethodDecl),
<CKToken: 0x6000000852d0>: Identifier[3:9] flashMessage (ObjCInstanceMethodDecl),
<CKToken: 0x6000000851e0>: Punctuation[3:21] : (ObjCInstanceMethodDecl),
<CKToken: 0x600000085190>: Punctuation[3:22] ( (ObjCInstanceMethodDecl),
<CKToken: 0x600000085140>: Identifier[3:23] NSString (ObjCClassRef),
<CKToken: 0x6000000850a0>: Punctuation[3:32] * (ParmDecl),
<CKToken: 0x6000000850f0>: Punctuation[3:33] ) (ParmDecl),
<CKToken: 0x600000085000>: Identifier[3:34] message (ParmDecl),
<CKToken: 0x600000085050>: Punctuation[3:41] ; (ObjCInstanceMethodDecl),
<CKToken: 0x600000084fb0>: Punctuation[4:1] @ (ObjCInterfaceDecl),
<CKToken: 0x600000084f60>: Keyword[4:2] end (ObjCInterfaceDecl)
)

I'd be more than happy to issue a pull request to fix this, but I'm not entirely sure where to start. This seems hard to fix because any arbitrary .h file could be C, C++, Obj-C or even Obj-C++.

Hi, I'm trying to build ClangKit by checking out the repo and running make from ClangKit/LLVM folder

Once compilation is finished (with no visible errors) I'm opening ClangKit.xcodeproj and straightaway it has libclangRewriteCore.a missing both from Libraries/Release and Libraries/Debug groups (as well as from the compiled files on the disk).

Any ideas what might cause this issues?

Thank you.

I'm getting a bunch of linker errors when compiling ClangKit.
I ran make in the LLVM directory. Is there anything else I need to do to get these problems solved?
By the way I'm on Xcode 5.1 and OS X 10.9.

I'm getting a bunch of linker errors when compiling ClangKit.
I ran make in the LLVM directory. Is there anything else I need to do to get these problems solved?
By the way I'm on Xcode 5.0.1 and OS X 10.9.

Compressing objects: 100% (254/254), done. Writing objects: 100% (450/450), 386.98 MiB | 45 KiB/s, done. Total 450 (delta 192), reused 436 (delta 184) remote: Error code: ccfbbe687a6acf7a75bcd9a65554d6cd remote: warning: Error GH413: Large files detected. remote: warning: See http://git.io/iEPt8g for more information. remote: warning: File ClangKit/LLVM/Debug/lib/libclang.a is 59.75 MB; this is larger than GitHub's recommended maximum file size of 50 MB remote: warning: File ClangKit/LLVM/Debug/lib/libclangARCMigrate.a is 92.75 MB; this is larger than GitHub's recommended maximum file size of 50 MB remote: warning: File ClangKit/LLVM/Debug/lib/libclangStaticAnalyzerCore.a is 91.92 MB; this is larger than GitHub's recommended maximum file size of 50 MB remote: error: File ClangKit/LLVM/Debug/lib/libclangAST.a is 123.01 MB; this exceeds GitHub's file size limit of 100 MB remote: error: File ClangKit/LLVM/Debug/lib/libclangSema.a is 191.36 MB; this exceeds GitHub's file size limit of 100 MB remote: error: File ClangKit/LLVM/Debug/lib/libLLVMCodeGen.a is 101.46 MB; this exceeds GitHub's file size limit of 100 MB remote: error: File ClangKit/LLVM/Debug/lib/libclangCodeGen.a is 147.11 MB; this exceeds GitHub's file size limit of 100 MB remote: error: File ClangKit/LLVM/Debug/lib/libclangStaticAnalyzerCheckers.a is 210.42 MB; this exceeds GitHub's file size limit of 100 MB To https://github.com/mralexgray/ClangKit.git ! [remote rejected] master -> master (pre-receive hook declined)

I parse .m files written in Objective-C using ClangKit, when the source code is about over 300 lines, I find out that the source code is truncated and no tokens about the missing part, the snapshot bellow shows the problem.

If the source code is not too long, it works fine.
I have check out the source code, but can't find any problem.
I am very confused about this, can any one help? Thanks a lot.

@" NSString * x;\n" ok so we know where this is...
"<CKToken: 0x100b1aed0>: Identifier[9:5] NSString

Let's try a completion on the next line...
@" NSStrin\n"

NSArray *comp = [tu completionResultsForLine:10 column:12];
NSLog(@"count: %ld", comp.count);

Result with Foundation.h linked.. 33718 without... 232.
That seems like ALL the completions available, to me... Not just those that correspond to this particular spot...

How can we get the results to correspond to the actual request / context?

On compilation, the code entered in the Test Target is returning an error in the Diagnostic as follows: "Foundation/Foundation.h file not found".
It seems that the Directives are not getting included

Great library, hopefully I can get this to work with an iOS project, But, I'm having some issue . I've run make in the LLVM directory, according to the tutorial, then I open the included project to build the framework, but I get some errors such as

Undefined symbols for architecture x86_64:
"llvm::SpecialCaseList::create(llvm::StringRef, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&)", referenced from:
clang::driver::SanitizerArgs::SanitizerArgs(clang::driver::ToolChain const&, llvm::opt::ArgList const&) in libclangDriver.a(SanitizerArgs.o)