Results 1 to 9 of 9

Thread: So, anyone got the replay parser working with java yet?

  1. #1

    So, anyone got the replay parser working with java yet?

    I'm not really good at c++ and would like to be able to parse dem files on a linux machine, so I have to go with the java version.

    After setting up all the protobuf stuff for java, I finally got together a java file which should work in theory, but throws an exception:

    Code:
    com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
    	at com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:68)
    	at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
    	at protobuf.Demo$CDemoFileInfo$Builder.mergeFrom(Demo.java:3113)
    	at protobuf.Demo$CDemoFileInfo$Builder.mergeFrom(Demo.java:1)
    	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300)
    	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
    	at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:202)
    	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:738)
    	at protobuf.Demo$CDemoFileInfo.parseFrom(Demo.java:2911)
    	at main.Application.main(Application.java:45)
    I'm not even sure if this is a general problem with my protobuf setup or if it's related to dem files/the dem parser.

    Here the java code snippet:

    Code:
    package main;
    
    import java.io.BufferedReader;
    [...]
    import org.xerial.snappy.Snappy;
    
    import com.google.protobuf.Descriptors.FieldDescriptor;
    
    import protobuf.Demo;
    import protobuf.Demo.CDemoFileInfo;
    
    public class Application {
    
    	/**
    	 * @param args
    	 */
    	public static void main(String[] args) {
    		
    		
    		try {
    			// Read dem file to string
    			FileInputStream ins = new FileInputStream("demo.dem");
    			DataInputStream in = new DataInputStream(ins);
    			BufferedReader br = new BufferedReader(new InputStreamReader(in));
    			String s;
    			StringBuffer sb = new StringBuffer();
    			while ((s = br.readLine()) != null)   {
    				sb.append(s);
    			}
    			in.close();
    
    			/* the byte-way
    			byte[] buffer = new byte[9999999];
    			FileInputStream is = new FileInputStream("demo.dem");
    			int lenbytes = is.read(buffer);
    			ByteArrayInputStream stream = new ByteArrayInputStream(buffer, 0, lenbytes);
    			*/
    			
    			// maybe uncompress?
    			String uncompressed;
    			if (Snappy.isValidCompressedBuffer(sb.toString().getBytes())) {
    				System.out.println("Uncompressing file...");
    				uncompressed = Snappy.uncompressString(sb.toString().getBytes());
    			} else {
    				uncompressed = sb.toString();
    			}
    			
    			// setup prot class and dump all fields			
    			CDemoFileInfo demo = Demo.CDemoFileInfo.parseFrom(uncompressed.getBytes());
    			Map<FieldDescriptor, Object> map = demo.getAllFields();
    			Set<FieldDescriptor> keys = map.keySet();
    			
    			FieldDescriptor next;
    			while ((next = keys.iterator().next()) != null) {
    				System.out.println(next.toString()+": "+map.get(next).toString());
    			}
    
    		} catch (FileNotFoundException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		} catch (IOException e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}
    	}
    
    }
    Anything wrong with that?

  2. #2
    Basic Member
    Join Date
    Dec 2011
    Posts
    8
    That's not really how the file is put together. If you look at demofile.cpp in the valve source, you can see that the first thing they do when they open the file is read a demo header from it, which is made up of an 8 byte char array followed by a 32-bit int (this struct, protodemoheader_t, is found in demofile.h). They don't actually start worrying about compression etc until they start reading messages out of the demo.

    Here's my python sketches to date: https://github.com/veryhappythings/p...r/parse.py#L45 - hopefully that makes a little sense. It gets as far as the file header. I don't think any of the compression stuff is anywhere close to right yet, just ignore that.

    The file structure is a DemoHeader followed by a series of message types and messages. You read the type, then the message. Read through demofiledump.cppoDump to see how it reads each type, then handles the message appropriately.

  3. #3
    Basic Member
    Join Date
    Feb 2012
    Posts
    6
    I dont't have the code in front of me right now, but the format goes something like this:

    Header [16 bytes]
    Commands [variable length]

    Each command is read as follows:
    Cmd [1-4 bytes] - also has bits set to indicate whether the data is compressed
    Tick [1-4 byes]
    Size (compressed) [1-4 bytes]

    If the data is compressed, decompresss it (from the next <size> bytes) using snappy. If not, do nothing.

    The next data is your protocol buffer message, and should be parsed using the protocol buffer classes.

    Note that there are exceptions for the demo header, FullPacketMessages and others. I suggest you take another look at the C++ files

  4. #4
    Quote Originally Posted by Flayra View Post
    I dont't have the code in front of me right now, but the format goes something like this:

    Header [16 bytes]
    Commands [variable length]

    Each command is read as follows:
    Cmd [1-4 bytes] - also has bits set to indicate whether the data is compressed
    Tick [1-4 byes]
    Size (compressed) [1-4 bytes]

    If the data is compressed, decompresss it (from the next <size> bytes) using snappy. If not, do nothing.

    The next data is your protocol buffer message, and should be parsed using the protocol buffer classes.

    Note that there are exceptions for the demo header, FullPacketMessages and others. I suggest you take another look at the C++ files
    That helped me a bit, but are you sure those numbers are correct? I had the impression, the header is 12 bytes not 16, because:

    Code:
    struct protodemoheader_t
    {
    	char demofilestamp[ 8 ]; // PROTODEMO_HEADER_ID
    	int32 fileinfo_offset;
    };
    = 8 byte + 4 byte (int32 has 4 bytes) = 12 bytes. and that's all which is read from the stream when filling this struct.

    Right now, I seem to only get junk out from the data:

    Code:
    	private static int readFileHeader(InputStream is, int pPos) throws IOException {
    		byte[] demofilestamp = new byte[16];
    		int lenbytes = is.read(demofilestamp, pPos, 16);
    		
    		pPos = lenbytes;
    		
    		System.out.println("Read-in Fileheader: "+byteArrayToString(demofilestamp));
    		
    		return pPos;
    	}
    
    	private static int readPackageType(InputStream is, int frameNumber) throws IOException {
    		int cmd = 0;
    		int tick = 0;
    		int compressed;
    		
    		DataInputStream dis = new DataInputStream(is);
    		cmd = dis.readInt();
    		tick = dis.readInt();
    		
    		compressed = cmd & DEM_IsCompressed; // DEM_IsCompressed = 112
    		cmd = (cmd & ~DEM_IsCompressed);
    		
    		System.out.println("Read package-head: "+Integer.toHexString(cmd)+", "+tick+" with compressed: "+compressed);
    
    		return cmd;
    	}
    Calling readFileHeader, then readPackageType once right after one another prints:



    While compressed seems to be a useful value (i suspect it defining the size of the compressed string, uncompressed when 0) but the tick value is just... weird. What does it do/say and why is it sometimes negative with huge values?

  5. #5
    Basic Member
    Join Date
    Feb 2012
    Posts
    6
    The header is 12 bytes as you say. However, you can't use readInt(), because that always reads 4 bytes, and that is not always correct. Check the ReadVarInt32 function in Demofile.cpp to see how it's done. The tick is a representation of the gametime. I believe it's 30 ticks per second, but I am not 100% sure.

    Compressed should also be seen as a boolean - if it's non-zero, the data for the current command is compressed. Otherwise it's not, simple as that.

  6. #6
    Yes, the compressed took me a while to understand what the !! were actually for. (the same as $x > 0)

    I recreated the ReadVarInt32 function, I'm not 100% what it actually does other than discarding some bits, but at least the numbers it returns make more sense now.

    Code:
    	private static int readVarInt32(byte[] buf, int index) {
    		int b;
    		int count = 0;
    		int result = 0;
    		
    		do {
    			if (count == 5) {
    				throw new IllegalArgumentException("Corrupt data");
    			} else if (index >= buf.length) {
    				throw new IndexOutOfBoundsException("index higher than buffer size!");
    			}
    			
    			b = buf[index++];
    			result |= ( b & 0x7F ) << (7 * count);
    			++count;
    		} while ((b & 0x80) > 0);
    		
    		return result;
    	}
    with the readMessageType function:
    Code:
    	private static int readMessageType(InputStream is, RefInteger tick) throws IOException {
    		int head;
    		int cmd;
    		byte[] buffer;
    		boolean compressed;
    		DataInputStream dis = new DataInputStream(is);
    
    		buffer = new byte[4];
    		dis.read(buffer);
    
    		head = readVarInt32(buffer, 0);
    			System.out.println("Read-in: "+byteArrayToString(buffer)+", after conversion: "+Integer.toHexString(head));
    		
    		compressed = (head & DEM_IsCompressed) > 0;
    			System.out.println("Compressed: "+Boolean.toString(compressed));
    		
    		cmd = (head & ~DEM_IsCompressed);
    			System.out.println("cmd: "+Integer.toHexString(head)+" & "+Integer.toHexString(~DEM_IsCompressed)+" = "+Integer.toHexString(cmd));
    		
    		buffer = new byte[4];
    		dis.read(buffer);
    		tick.setValue(readVarInt32(buffer, 0));
    			System.out.println("tick: "+Integer.toHexString(tick.getValue()));
    
    		//System.out.println("Read package-head: "+Integer.toHexString(cmd)+", "+tick+" with compressed: "+compressed);
    
    		return cmd;
    	}
    where the buffer is the stream in the c++ implementation. I see that it makes sense to use the index as a permanent offset pointer, however I'm positive that you don't need this in java. (Correct me if I'm wrong, but every time you pull data from the stream, the pointer shifts itself to the next byte(s), right?)

    So with this seeming to work now, I checked the first Package in the c++ and in my java implementation on the same demo file.
    Result was, that my java tool returned as first values:

    Read-in Fileheader: PBUFDEM ??
    Pos after header is now: 12
    Read-in: [byte junk], after conversion: 1
    Compressed: false
    cmd: 1 & ffffff8f = 1
    tick: 8
    The c++ implementation with some more debug output says:

    xxx: Original CMD[4]: 1 from ReadVarInt32(1) ~mask ffffff8f cmd[4]: 1, tick[4]: 0 -- DoDump tick 0, DemoCommmand: 1
    xxx: Original CMD[4]: 8 from ReadVarInt32(97) ~mask ffffff8f cmd[4]: 8, tick[4]: 0 -- DoDump tick 0, DemoCommmand: 8
    Interesting to note here is, that the tick value in the c++ tool remain at 0 until the >14th package, where the java tool already drops tick = 8 in the first package after the overall file header.

    Anything wrong with the code yet? Maybe an error on how I pull the data from the stream?

  7. #7
    Basic Member
    Join Date
    Dec 2011
    Posts
    4
    Your treatment of the VarInt32 is still not quite right. It is a variable length integer, so it can be stored in anywhere between 1 and 4 bytes depending on how big the integer actually is. If the highest bit (The & 0x80) is set, then that indicates that there is another byte that follows. If this byte isn't set then it is the end of the integer. That means your readVarInt32 function is basically correct but you are reading in 4 bytes every time when you should only be reading in as much data as required by the VarInt32. Here's a piece of code that is wrong:

    DataInputStream dis = new DataInputStream(is);

    buffer = new byte[4];
    dis.read(buffer);
    head = readVarInt32(buffer, 0);
    You will need to read in one byte at a time, and then only read another if this bit flag is actually set. I didn't go through the rest of your code that closely, but I suspect all of your issues are coming from the incorrect handling of the variable length ints.

  8. #8
    YES! That was it.

    Now I'm able to read the packages and use the data. Thx!

  9. #9
    Basic Member xXAVx's Avatar
    Join Date
    Apr 2012
    Posts
    43
    Hello! Would you be willing to provide your fixed Java code?
    I am developing my own parser, so I would greatly appreciate some working code.

    Thanks!

    edit:
    Thanks for the code! Although, answering myself: it is actually trivial to write Java code by mimic'ing the functions in demofile.cpp and demofileDump.cpp. It just takes a handful of hours to get the stuff going. I will post my end result in Java once I get something I am satisfied with. If you are "desperate" for Java code, you can send me a private message me, I will be happy to help.
    Last edited by xXAVx; 08-09-2012 at 02:15 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •