Mercurial > jhg
annotate design.txt @ 115:c0cc2535462c
Introduced channels to pipeline (and easily filter) data streams
author | Artem Tikhomirov <tikhomirov.artem@gmail.com> |
---|---|
date | Thu, 03 Feb 2011 23:32:08 +0100 (2011-02-03) |
parents | d55d4eedfc57 |
children | 44b97930570c |
rev | line source |
---|---|
1
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
1 FileStructureWalker (pass HgFile, HgFolder to callable; which can ask for VCS data from any file) |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
2 External uses: user browses files, selects one and asks for its history |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
3 Params: tip/revision; |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
4 Implementation: manifest |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
5 |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
6 Log --rev |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
7 Log <file> |
2
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
8 HgDataFile.history() or Changelog.history(file)? |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
9 |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
10 |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
11 Changelog.all() to return list with placeholder, not-parsed elements (i.e. read only compressedLen field and skip to next record), so that |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
12 total number of elements in the list is correct |
1
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
13 |
a3576694a4d1
Repository detection from local/specified directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
diff
changeset
|
14 hg cat |
2
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
15 Implementation: logic to find file by name in the repository is the same with Log and other commands |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
16 |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
17 |
08db726a0fb7
Shaping out low-level Hg structures
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
1
diff
changeset
|
18 Revlog |
4
aa1912c70b36
Fix offset issue for inline revlogs. Commandline processing.
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
2
diff
changeset
|
19 What happens when big entry is added to a file - when it detects it can't longer fit into .i and needs .d? Inline flag and .i format changes? |
aa1912c70b36
Fix offset issue for inline revlogs. Commandline processing.
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
2
diff
changeset
|
20 |
22
603806cd2dc6
Status of local working dir against non-tip base revision
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
21
diff
changeset
|
21 What's hg natural way to see nodeids of specific files (i.e. when I do 'hg --debug manifest -r 11' and see nodeid of some file, and |
603806cd2dc6
Status of local working dir against non-tip base revision
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
21
diff
changeset
|
22 then would like to see what changeset this file came from)? |
4
aa1912c70b36
Fix offset issue for inline revlogs. Commandline processing.
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
2
diff
changeset
|
23 |
aa1912c70b36
Fix offset issue for inline revlogs. Commandline processing.
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
2
diff
changeset
|
24 ---------- |
6
5abe5af181bd
Ant script to build commands and run sample
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
5
diff
changeset
|
25 + support patch from baseRev + few deltas (although done in a way patches are applied one by one instead of accumulated) |
4
aa1912c70b36
Fix offset issue for inline revlogs. Commandline processing.
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
2
diff
changeset
|
26 + command-line samples (-R, filenames) (Log & Cat) to show on any repo |
6
5abe5af181bd
Ant script to build commands and run sample
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
5
diff
changeset
|
27 +buildfile + run samples |
9
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
28 *input stream impl + lifecycle. Step forward with FileChannel and ByteBuffer, although questionable accomplishment (looks bit complicated, cumbersome) |
14
442dc6ee647b
Show correct time
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
11
diff
changeset
|
29 + dirstate.mtime |
43
1b26247d7367
Calculate result length of the patch operarion, when unknown
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
41
diff
changeset
|
30 +calculate sha1 digest for file to see I can deal with nodeid. +Do this correctly (smaller nodeid - first) |
18
02ee376bee79
status operation against current working directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
17
diff
changeset
|
31 *.hgignored processing |
25
da8ccbfae64d
Reflect Nodeid's array is exactly 20
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
22
diff
changeset
|
32 +Nodeid to keep 20 bytes always, Revlog.Inspector to get nodeid array of meaningful data exact size (nor heading 00 bytes, nor 12 extra bytes from the spec) |
26
71a9ba42cee8
Memory-mapped files for bigger files. Defect reading number of bytes greater than size of the buffer fixed
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
25
diff
changeset
|
33 +DataAccess - implement memory mapped files, |
49
26e3eeaa3962
branch and user filtering for log operation
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
43
diff
changeset
|
34 +Changeset to get index (local revision number) |
60
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
35 +RevisionWalker (on manifest) and WorkingCopyWalker (io.File) talking to ? and/or dirstate (StatusCollector and WCSC) |
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
36 +RevlogStream - Inflater. Perhaps, InflaterStream instead? branch:wrap-data-access |
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
37 +repo.status - use same collector class twice, difference as external code. add external walker that keeps collected maps and use it in Log operation to give files+,files- |
78
c25c5c348d1b
Skip metadata in the beginning of a file content. Parse metadata, recognize copies/renames
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
64
diff
changeset
|
38 + strip \1\n metadata out from RevlogStream |
84
08754fce5778
updated design questions
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
78
diff
changeset
|
39 + hash/digest long names for fncache |
60
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
40 |
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
41 delta merge |
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
42 DataAccess - collect debug info (buffer misses, file size/total read operations) to find out better strategy to buffer size detection. Compare performance. |
41
858d1b2458cb
Check integrity for bundle changelog. Sort nodeids when calculating hash
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
33
diff
changeset
|
43 |
78
c25c5c348d1b
Skip metadata in the beginning of a file content. Parse metadata, recognize copies/renames
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
64
diff
changeset
|
44 Strip off metadata from beg of the stream - DataAccess (with rebase/moveBaseOffset(int)) would be handy |
93
d55d4eedfc57
Switch to Path instead of String in filenames returned by various status operations
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
84
diff
changeset
|
45 Parameterize StatusCollector to produce copy only when needed. And HgDataFile.metadata perhaps should be moved to cacheable place? |
60
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
46 |
18
02ee376bee79
status operation against current working directory
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
17
diff
changeset
|
47 Status operation from GUI - guess, usually on a file/subfolder, hence API should allow for starting path (unlike cmdline, seems useless to implement include/exclide patterns - GUI users hardly enter them, ever) |
60
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
48 -> recently introduced FileWalker may perhaps help solving this (if starts walking from selected folder) for status op against WorkingDir? |
9
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
49 |
84
08754fce5778
updated design questions
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
78
diff
changeset
|
50 ? Can I use fncache (names from it - perhaps, would help for Mac issues Alex mentioned) |
08754fce5778
updated design questions
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
78
diff
changeset
|
51 ? Does fncache lists both .i and .d (iow, is it true hashed <long name>.d is different from hashed <long name>.i) |
08754fce5778
updated design questions
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
78
diff
changeset
|
52 |
15
865bf07f381f
Basic hgignore handling
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
14
diff
changeset
|
53 ??? encodings of fncache, .hgignore, dirstate |
16
254078595653
Print manifest nodeid
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
15
diff
changeset
|
54 ??? http://mercurial.selenic.com/wiki/Manifest says "Multiple changesets may refer to the same manifest revision". To me, each changeset |
254078595653
Print manifest nodeid
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
15
diff
changeset
|
55 changes repository, hence manifest should update nodeids of the files it lists, effectively creating new manifest revision. |
15
865bf07f381f
Basic hgignore handling
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
14
diff
changeset
|
56 |
64
19e9e220bf68
Convenient commands constitute hi-level API. org.tmatesoft namespace, GPL2 statement
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
61
diff
changeset
|
57 ? hg status, compare revision and local file with kw expansion and eol extension |
19e9e220bf68
Convenient commands constitute hi-level API. org.tmatesoft namespace, GPL2 statement
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
61
diff
changeset
|
58 ? subrepos in log, status (-S) and manifest commands |
19e9e220bf68
Convenient commands constitute hi-level API. org.tmatesoft namespace, GPL2 statement
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
61
diff
changeset
|
59 |
93
d55d4eedfc57
Switch to Path instead of String in filenames returned by various status operations
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
84
diff
changeset
|
60 |
64
19e9e220bf68
Convenient commands constitute hi-level API. org.tmatesoft namespace, GPL2 statement
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
61
diff
changeset
|
61 Commands to get CommandContext where they may share various caches (e.g. StatusCollector) |
93
d55d4eedfc57
Switch to Path instead of String in filenames returned by various status operations
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
84
diff
changeset
|
62 Perhaps, abstract classes for all Inspectors (i.e. StatusCollector.Inspector) for users to use as base classes to protect from change? |
64
19e9e220bf68
Convenient commands constitute hi-level API. org.tmatesoft namespace, GPL2 statement
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
61
diff
changeset
|
63 |
9
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
64 >>>> Effective file read/data access |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
65 ReadOperation, Revlog does: repo.getFileSystem().run(this.file, new ReadOperation(), long start=0, long end = -1) |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
66 ReadOperation gets buffer (of whatever size, as decided by FS impl), parses it and then reports if needs more data. |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
67 This helps to ensure streams are closed after reading, allows caching (if the same file (or LRU) is read few times in sequence) |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
68 and allows buffer management (i.e. reuse. Single buffer for all reads). |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
69 Scheduling multiple operations (in future, to deal with writes - single queue for FS operations - no locks?) |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
70 |
60
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
71 WRITE: Need to register instances that cache files (e.g. dirstate or .hgignore) to FS notifier, so that cache may get cleared if the file changes (i.e. WriteOperation touches it). |
613c936d74e4
Log operation to output mode detailed (added, removed) files
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
55
diff
changeset
|
72 |
9
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
73 File access: |
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
74 * NIO and mapped files - should be fast. Although seems to give less control on mem usage. |
21
e929cecae4e1
Refactor to move revlog content to base class
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
20
diff
changeset
|
75 * Regular InputStreams and chunked stream on top - allocate List<byte[]>, each (but last) chunk of fixed size (depending on initial file size) |
9
d6d2a630f4a6
Access to underlaying file data wrapped into own Access object, implemented with FileChannel and ByteBuffer
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
6
diff
changeset
|
76 |
26
71a9ba42cee8
Memory-mapped files for bigger files. Defect reading number of bytes greater than size of the buffer fixed
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
25
diff
changeset
|
77 <<<<< |
71a9ba42cee8
Memory-mapped files for bigger files. Defect reading number of bytes greater than size of the buffer fixed
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
25
diff
changeset
|
78 |
71a9ba42cee8
Memory-mapped files for bigger files. Defect reading number of bytes greater than size of the buffer fixed
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
25
diff
changeset
|
79 Tests: |
61
fac8e7fcc8b0
Simple test framework - capable of parsing Hg cmdline output to compare with Java result
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
60
diff
changeset
|
80 DataAccess - readBytes(length > memBufferSize, length*2 > memBufferSize) - to check impl is capable to read huge chunks of data, regardless of own buffer size |
fac8e7fcc8b0
Simple test framework - capable of parsing Hg cmdline output to compare with Java result
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
60
diff
changeset
|
81 |
fac8e7fcc8b0
Simple test framework - capable of parsing Hg cmdline output to compare with Java result
Artem Tikhomirov <tikhomirov.artem@gmail.com>
parents:
60
diff
changeset
|
82 ExecHelper('cmd', OutputParser()).run(). StatusOutputParser, LogOutputParser extends OutputParser. construct java result similar to that of cmd, compare results |